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Research  conducted  by  the  Systems  Research  Group  under  Contract 
DAAH01-71-C-1258  is  described  in  this  report.  The  report  is  organized 
so  that  the  Filter  simulation  experimental  concept  is  introduced  in 
Chapter  1,  and  the  Filter  experimental  models  are  described  in  Chapter  2. 
Chapter  3 presents  the  experimental  results  and  an  analysis  of  these 
results  to  assess  the  validity  and  efficiency  of  the  Filter  concept. 
Chapter  4 presents  the  conclusions  drawn  from  the  research.  Detailed 
mathematical  derivations  and  computer  program  descriptions  are  presented 
in  the  Appendixes. 

Although  this  is  a final  report  for  the  Filter  Model  contract, 
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and  participation  in  military  operations  research. 
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neering, The  Ohio  State  University,  and  should  not  be  considered  as 
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pressed or  Implied,  until  reviewed  and  evaluated  by  those  agencies  and 
subsequently  endorsed. 

The  cooperation  received  from  MI  COM  personnel  in  the  conduct  of 
this  research  has  been  extremely  helpful.  In  particular,  we  wish  to 
acknowledge  the  advice  and  assistance  provided  by  Miss  Elizabeth 
Watkins  and  Messrs,  James  Stage  and  Ernest  Petty. 

In  addition,  we  would  like  to  acknowledge  the  important  contribu- 
tions of  Mr.  Robert  Wilhelm  who  wrote  the  Filter  Model  computer  programs 
used  in  this  research. 


ii 


TABLE  OF  CONTENTS 


OIIA1TER  PAGE 


1 THE  FILTER  CONCEPT  OF  SIMULATION  EXPERIMENTS  ....  1 

Introduction  .....  ..  1 

Use  of  Filter  Model 1 

Review  of  Relevant  Material  .....  2 

Research  Required  for  the  Filter  Model  ....  3 

Organization  of  the  Report  ..........  4 

2 FILTER  MODEIS 5 

Example  of  Experimenting  With  Blocks 5 

Filter  Model  Versions  9 

HE  VIE  Model 12 

HEVCE  Model l4 

Least -Cost  Experimental  Design 18 

3 EXPERIMENTAL  RESULTS 21 

Introduction  21 

Validity  of  the  Filter  Models 26 

Cost  of  Estimating  Mean  Systesm  Performance  . . 29 

Cost  Comparisons  for  Achieving  Equivalent 

Variances  31 

Cost  Comparisons  for  Reducing  Variance  Below 
Specified  Upper  Limits  ...........  34 

Sensitivity  of  Experimental  Plans  to  Input 

Variability  37 

Cost  of  Comparing  Mean  System  Performance  ...  4l 

Validity  of  the  Expression  for 
V(X, ) 4l 

Cost  of  Estimating  Differences  in  Average 

System  Performance  .............  43 

Screening  Candidate  Alternatives  ...•••»  44 

4 SUMMARY  AND  CONCLUSIONS 47 

APPENDIXES 

A DERIVATION  OF  ESTIMATORS  FOR  FILTER  MODEL  PARAMETERS 

(HEVCE  MODEL) 49 

Introduction  49 

Filter  Model  Structure  .....  49 

iii 


TABLE  OF  CONTENTS  (continued) 


APPENDIX  PAGE 


A Variance  of  Estimator  for  Mean  System 

Performance  . . 53 

Estimator  for  ean  System  Performance  . 53 

Variance  of  Meen  System  Performance 
Estimator 5^ 

Estimation  of  the  Variance  of  the  Estimator 
of  System  Mean  Performance  ........  57 

Variance  of  Estimator  for  Differences  in 

Mean  Surface  Performance  6l 

Estimation  of  Replication  Variance  ...  66 

B LEAST-COST  EXPERIMENTAL  DESIGN  ALGORITHM 69 


C 


DERIVATION  OF  ESTIMATORS  FOR  THE  FILTER  INDEPENDENT 
EFFECTS  AND  HETEROGENEOUS  VARIANCES  MODEL  (HEVIE)  77 


Model  Description  ...... 

Estimators . 

Replication  Variances  . . 
Block  Variance  ...... 

Block  Sum  of  Squares  . * * 
Interaction  Sum  of  Squares 
Block  Variance  Estimator  . 


77 

78 
78 

78 

79 

80 
82 


D DERIVATION  OF  ESTIMATORS  FOR  THE  FILTER  INDEPENDENT 


EFFECTS  AND  HOMOGENEOUS  VARIANCES  MODEL  (HOVIE).  85 

Model  Description  85 

Parameter  Estimators  . 86 

E FILTER  PROGRAM 89 

Purpose  89 

Input  Data  Format  89 

Definition  of  Variables 90 

Method 91 

References  93 


iv 


LIST  OF  TABLES 


Table  No,  Page 

2.1  Red  Casualty  Data  7 

3.1  Red  Casualty  Data  from  Blocked  Simulation  Repli- 

cations   23 

3.2  Rounds  Fired  at  Blue  Weapons  Data  from  Blocked 

Simulation  Replications  24 

3.3  Blue  Exposure  Time  Data  from  Blocked  Simulation 

Replications  2k 

3.4  Engagement  Range  Data  from  Blocked  Replications  . 25 

3.5  System  1 Performance  Data  from  Independent  Full- 

Length  Simulation  Replications 25 

3.6  Estimates  of  the  Variance  of  Total  Red  Casualties 

for  One  Simulation  Replication 27 

3.7  Estimates  of  the  Variance  of  Total  Pounds  Fired 

at  Blue  for  One  Simulation  Replication  ....  27 

3.8  Estimates  of  the  Variance  of  Blue  Exposure  Time  . 28 

3.9  Estimates  of  the  Variance  of  Engagement  Range  . . 28 

3.10  Cost  Comparison  to  Achieve  Equivalent  Variance  of 

Average  Red  Casualties,  Two  Blocks  Versus  Inde- 
pendent Full-Length  Runs  ...........  32 

3.11  Cost  Comparison  to  Achieve  Equivalent  Variance  of 

Rounds  Fired  at  Blue,  Two  Blocks  Versus  Inde- 
pendent Full-Length  Runs  32 

3.12  Cost  Comparison  to  Achieve  Equivalent  Variance  of 

Blue  Exposure  Time,  Two  Blocks  Versus  Independ- 
ent Full-Length  Runs  33 

3.13  Cost  Comparison  to  Achieve  Equivalent  Variance  of 

Engagement  Range,  Two  Blocks  Versus  Independent 
Full-Length  Runs  33 

3.14  Cost  Comparison  to  Achieve  Average  Red  Casualties 

Variance  Lower  Than  1,00  - Additional  Replica- 
tions After  Two  Blocks  Versus  Independent  Full- 
Length  Runs  ..........  35 

# 

> 


v 


LIST  OF  TABLES  (continued) 


Table  No«  Page 

3.15  Cost  Comparison  to  Achieve  Average  Rounds  Fired 

Variance  Lower  Than  1,20  - Additional  Replica- 
tions After  Two  Blocks  Versus  Independent  Full- 
Length  Runs  35 

3.16  Cost  Comparison  to  Achieve  Average  Blue  Exposure 

Time  Variance  Lower  Than  3150  - Additional 
Replications  After  Two  Blocks  VerBus  Independ- 
ent Full-Length  Runs  . 36 

3.17  Cost  Comparison  to  Achieve  Average  Engagement  Range 

Variance  Lower  Than  350  - Additional  Replications 
After  Two  Blocks  Versus  Independent  Full-Length 
Runs  ...........  36 

3.18  Comparison  of  the  Estimates  of  HEVCE  Parameters 

from  Two  and  Four  Blocks  of  Data  ........  38 

3.19  Comparison  of  Least-Cost  Experimental  Plans  Sub- 

sequent to  Two  Blocks 39 

3.20  Comparison  of  Least-Cost  Experimental  Plans  Sub- 

sequent to  Two  Blocks  40 

3.21  Variance  of  the  Difference  in  Average  Performances 

Validity  Check 42 

3.22  Variance  of  the  Difference  in  Average  Performance  . 43 

3.23  Efficiency  of  a Single  Block  Filter  With  Five 

Replications  ............  45 

LIST  OF  FIGURES 

Figure  No.  Page 


B.l  Algorithm  for  Determining  Least-Cost  Solution  ...  73 


vi 


I 


CHAPTER  1 

THE  FILTER  CONCEPT  OF  SIMULATION  EXPERIMENTS 


Introduction 

Research  conducted  under  contract  to  the  United  States  Anny  Missile 
Command  (MICOM)  by  The  Ohio  State  University  Systems  Research  Group  has 
led  to  the  development  of  DYNCOM,  a high-resolution  model  of  land  combat 
capable  of  evaluating  the  performance  of  tactical  units  employing  ad- 
vanced missile  systems.  This  report  describes  the  results  of  research 
to  develop  methodology  for  applying  DYNCOM  in  a more  rapid  manner  and 
for  reducing  computer  usage  costs.  The  objective  of  this  research  is  to 
develop  a dynamic  simulation  filter  which  can  screen  candidates,  iden- 
tifying weaker  alternatives,  with  much  less  cost  than  operating  the  full- 
scale  DYNCOM  simulation. 
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Use  of  Filter  Models 


A study  of  weapon  design  characteristics  and  their  relationships 
with  tactical-unit  combat  effectiveness  often  requiries  the  evaluation 
of  many  weapon  system  design  alternatives  and  force  mixes.  Moreover, 
to  adequately  determine  the  relationships  between  individual  weapon 
performance  as  constrained  by  the  battlefield  environment  and  tactical 
unit  performance,  a high-resolution  simulation  such  as  DYNCOM  often  is 
needed.  This  high  resolution  model,  however,  can  be  costly  to  apply  if 
a large  number  of  alternatives  are  being  considered.  Current  simulation 
costs  for  DYNCOM  are  running  up  to  $200  for  a single  full-length  battle 
involving  armored  weapons,  artillery,  helicopters,  and  air  defense  weapons. 
To  reduce  the  cost,  lower  resolution  filter  models  and  subjective  analyses 
have  been  used  in  the  past  to  filter  candidate  weapons  and  force  mixes. 

An  example  is  provided  by  the  use  in  the  TATAWS  study  of  a simplified 
deterministic  simulation  called  FILTER  (Booz-Allen  Applied  Research, 

1967)  to  screen  candidate  weapons  and  weapon  mixes  to  reduce  the  work 
load  on  the  more  detailed  IUA  simulation  (USACDC,  1968).  However,  the 
lack  of  homogeneity  between  filter  and  simulation  results  in  the  TATAWS 
study  created  severe  problems  in  implementing  this  concept.  That  is, 
candidates  that  fared  poorly  in  the  FILTER  model  could  do  well  in  the 
IUA  simulation.  Moreover,  this  homogenity  problem  is  always  a potential 
problem  when  using  a separate  low-resolution  filter  model  to  screen  can- 
didates for  a more  detailed  simulation. 

The  research  presented  in  this  report  proposes  that  portions  of 
DYNCOM  itself  serve  as  a dynamic  filter  when  required  in  the  context 
of  the  military  problem  being  studied.  These  procedures  will  be  similar 
to  physical  experimental  design  in  that  DYNCOM  will  be  used  to  generate 
an  environment  for  comparison  among  the  alternatives  being  considered. 

This  environment  can  be  used  as  the  starting  point  for  further  simula- 
tion experiments  on  each  alternative.  The  agricultural  experimenter 
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who  tests  the  productivity  of  different  seed  types  by  planting  groups 
consisting  of  each  type  of  seed  at  different  points  in  a field  is  using 
a similar  concept  in  physical  experimentation.  Each  group  of  seeds  has 
a similar  environment,  and  the  results  from  each  group  are  correlated 
so  that  contrasts  between  the  seed  productivities  are  heightened.  Each 
group  is  called  a block  in  the  language  of  experimental  design. 

Use  of  DYNCOM  both  as  a fast  operating  filter  model  and  as  a model 
for  a full-scale  battle  assures  a greater  degree  of  homogeneity  between 
filter  and  simulation,  and  the  ability  to  construct  this  filter  model 
from  the  DYNCOM  program  takes  advantage  of  the  flexibility  already 
inherent  in  the  DYNCOM  program.  The  following  example  illustrates  the 
concepts  described  above. 

Assume  that  a study  is  being  conducted  to  evaluate  the  combat 
effectiveness  of  the  MAW  class  of  weapons  versus  the  LAW  and  HAW  classes 
of  weapons.  Further,  assume  that  the  first  problem  in  the  study  is  to 
determine  the  best  MAW  weapon,  where  various  MAW  systems  have  differing 
accuracy,  lethality,  and  rate  of  fire  characteristics.  To  start,  the 
procedure  would  specify  that  one  complete  simulation  of  DYNCOM  be  made. 
Simulation  check  points  would  be  established  to  determine  the  portion  of 
the  battle  during  which  the  different  MAW  candidates  would  be  firing.  A 
dynamic  filter  would  thus  be  determined  for  studying  the  different  MAW 
weapons  by  identifying  the  first  event  that  a MAW  weapon  would  be  firing. 
Inputs  to  the  Filter  Model  would  be  dynamic  in  that  they  would  be  deter- 
mined from  the  complete  simulation  using  DYNCOM' s restart  capability. 

One  set  of  inputs  would  establish  a block  using  the  physical  experimental 
design  terminology.  Replications  by  the  dynamic  filter  would  be  made 
with  filter  inputs  being  modified  as  required,  until  results  with  the 
required  statistical  precision  are  obtained.  Several  replications  in  a 
single  block  may  be  made  for  each  system  alternative  by  changing  the 
random  number  sequence.  Moreover,  identical  random  number  sequences 
may  be  used  for  each  replication  in  a given  block  to  increase  the  corre- 
lation among  results. 


Review  of  Relevant  Literature 


Application  of  these  concepts  to  a high  resolution  combat  simulation 
has  not  been  reported  in  the  available  literature.  However,  Fishman 
(1973)  and  Emshoff  and  Sisson  (1970)  suggest  that  these  concepts  may  be 
profitably  applied  to  increase  the  efficiency  of  simulation  experiments. 
Neither  authors  offer  any  examples  which  show  that  actual  simulation 
experiments  may  benefit  from  these  precedures. 

An  experimental  model  which  could  be  used  to  analyze  results  from 
blocked  experiments  is  called  the  Two-Way  Mixed  Model  (Winer,  1971; 
Graybill,  1961;  Hicks,  1964;  Scheffe,  1959;  and  Hocking,  1973).  In 
this  model  two  factors  are  present  and  many  levels  or  possible  values 
may  exist  for  each  factor.  In  this  application,  one  factor  is  the  combat 
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system  being  analyzed  and  each  particular  system  is  a different  level 
for  the  combat  system  factor.  This  factor  is  regarded  as  being  fixed 
or  deterministic  because  the  different  systems  are  predetermined  and 
not  randomly  selected.  The  block  environment  is  the  other  factor  and 
it  is  regarded  as  being  random  because  a very  large  number  of  possible 
block  environments  exists  and  each  one  is  randomly  selected  from  this 
population.  Usually  the  assumptions  are  made  that 

1.  all  random  effects  are  independent, 

2.  all  variances  are  constant  with  respect  to  the  system  being 
represented,  and 

3.  all  systems  have  the  same  number  of  replications  per  block. 

Scheffe'  (1959)  and  Hocking  (1973)  present  Two-Way  Mixed  Models  with 
correlations  among  random  effects;  however,  the  correlations  proposed  by 
their  models  assume  homogenous  variances  for  each  system  and  the 
covariances  among  the  replications  within  a block  are  assumed  to  be  zero. 
Since  data  from  DYNCOM  suggest  heterogeneous  variances  for  each  system 
and  the  experimental  procedure  may  introduce  correlations  among  the 
replications,  a filter  model  incorporating  correlations  will  have  to  be 
a different  model  than  that  developed  by  Scheffe  and  Hocking. 


Research  Required  for  the  Filter  Model 

As  noted  in  the  literature  review,  application  of  the  concepts  of 
blocked  experiments  to  simulation  experiments  has  not  been  performed. 
Moreover,  the  nature  of  the  combat  process  simulated  by  DYNCOM  indicates 
that  available  Two-Factor  Mixed  Models  may  be  inadequate  for  representing 
DYNCOM  results.  Accordingly  research  is  required  to  formulate  a model 
explicitly  designed  to  represent  DYNCOM  results,  and  an  algorithm  is 
required  for  determining  the  least-cost  test  plan  using  this  model.  Then 
this  model  and  other  currently  available  models  need  to  be  tested  with 
DYNCOM  results  to  determine  their  validity.  That  is,  the  estimates  of 
the  mean  and  variance  of  system  performance  must  be  homogeneous  when 
estimated  by  independent  full-length  DYNCOM  runs  and  blocked  filter  runs . 

In  addition,  blocked  data  from  DYNCOM  need  to  be  analyzed  to 
determine  whether  the  Filter  Model  actually  estimates  systems  performance 
more  economically  than  independent  full-length  DYNCOM  replications.  Two 
types  of  comparisons  need  to  be  made,  i.e.,  the  Filter  Model  needs  to 
be  compared  with  independent  full-length  replications  with  respect  to 
their  efficiency  in  estimating  mean  system  performance  and  with  respect 
to  their  efficiency  in  estimating  differences  in  mean  performance  between 
unlike  systems.  The  process  of  filtering  requires  comparisons  to  be 
made  between  different  systems  so  the  difference  in  average  performance 
between  two  systems  is  important.  Finally,  the  ability  of  the  Filter 
Model  to  screen  candidates  with  small  samples  or  modest  computer  expense 
is  required  to  be  tested  to  indicate  the  utility  of  the  filter  concept. 
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Organization  of  the  Report 


An  overview  of  the  three  Filter  Models  hypothesized  for  use  with 
DYNCOM  is  presented  in  Chapter  2.  In  addition,  estimators  for  model 
parameters  are  presented  in  Chapter  2.  Chapter  3 presents  the  analysis 
of  DYNCOM  results;  a valid  Filter  Model  is  identified  in  the  chapter, 
and  the  economies  accuring  from  use  of  the  Filter  Model  are  estimated. 
Chapter  4 presents  a summary  of  the  research  results  and  the  principal 
conclusions. 

Detailed  mathematical  derivations  are  presented  in  the  appendixes. 
Appendix  A presents  the  Filter  Model  explicitly  designed  to  include 
correlations  and  heterogeneous  variances  expected  to  occur  with  DYNCOM. 
Estimators  for  the  model  parameters  are  derived  in  Appendix  A.  The 
least-cost  experimental  design  algorithm  is  developed  in  Appendix  B. 
Appendix  C presents  a derivation  of  estimators  for  a version  of  the 
Filter  Model  incorporating  independent  effects  and  heterogeneous  variances. 
A derivation  of  estimators  for  another  Filter  Model  version  having  in- 
dependent effects  and  homogeneous  variances  is  presented  in  Appendix  D. 
Appendix  E presents  instructions  for  using  the  Filter  program  to  estimate 
model  parameters  and  calculate  least-cost  test  plans. 
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CHAPTER  2 
FILTER  MODELS 


Introduction 

As  described  in  Chapter  1,  the  basic  purpose  of  the  Filter  Model  is 
to  screen  alternative  systems  to  eliminate  less  desirable  systems  using 
replications  of  homogeneous  combat  environments  called  block  environments. 
In  this  chapter,  several  models  of  test  results  under  conditions  of  block- 
ing are  described  for  consideration  as  the  Filter  Model.  These  models 
are  used  to  predict  the  variance  of  mean  system  performance  estimates 
when  experiments  are  blocked.  Moreover,  an  important  consideration  is 
the  variance  of  the  difference  between  two  syBtesn  performance  estimates 
since  the  basic  purpose  of  the  Filter  Model  is  to  make  comparisons  among 
systems.  In  addition,  the  characteristics  of  estimators  for  model  param- 
eters required  to  implement  the  Filter  Models  are  presented.  Given 
parameter  estimates,  an  algorithm  specifying  the  least-cost  test  plan  to 
reduce  the  variance  of  the  mean  system  performance  estimator  below  a 
specified  upper  limit  is  presented. 
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In  order  to  more  clearly  present  the  concepts  inherent  in  the  dif- 
ferent Filter  Models,  a numerical  example  is  presented  in  the  following 
section.  After  this  example,  the  Filter  Models  are  presented  and  com- 
pared by  reference  to  the  example. 


Example  of  Experimenting  with  Blocks 

Consider  a combat  simulation  which  is  being  used  to  compare  the  per- 
formance of  armored  battalions  with  different  anti-tank  weapons  having 
dissimilar  accuracy,  lethality,  and  rate  of  fire  characteristics.  These 
systems  are  noted  as  System  1 and  System  2.  Of  course,  the  nti-tank 
weapons  in  question  are  only  one  of  several  weapons  in  the  armored 
battalions.  The  system  analyst  is  interested  in  estimating  performance 
measures  in  comparing  these  systems,  and  one  of  these  performance  mea- 
sures is  expected  enemy  casualties  during  a particular  engagement. 

In  the  simulation  experimentation  process,  blocks  are  generated  by 
simulating  the  initial  part  of  the  engagement  until  one  of  the  two  systems 
being  compared  can  be  employed  and  then  generating  a restart  data  set. 

This  data  set  specifies  the  complete  battle  situation  as  known  to  the 
simulation,  and  the  data  set  implies  a block  environment.  The  block 
environment  consists  of  the  state  of  each  combatant  as  indicated  by  the 
following  status  variables: 

1.  casualty  status;  i.e.,  mobility  kill,  fire  power 
kill,  complete  kill,  or  no  kill; 

2.  enemy  weapons  detected; 


5 


3.  position; 

4.  velocity; 

5.  ammunition  supply; 

6.  fire  support  missions  requested; 

7 . neutralization  or  suppression  state;  and 

8.  current  target,  etc. 

Blocks  are  replicated  by  simulating  the  remainder  of  the  battle  until 
its  conclusion  for  each  system  alternative  and  then  repeating  this  pro- 
cess with  a new  string  of  randan  numbers  starting  with  the  same  block 
environment. 

Two  options  exist  for  performing  the  replications.  One  option  con- 
sists of  using  a unique  sequence  of  random  numbers  for  each  system  and 
each  replication.  Another  option  introduces  more  correlation  among 
results  by  using  the  same  sequence  for  each  alternative  for  a given 
replication.  This  method  applies  the  simulation  variance  reduction 
concept  of  introducing  positive  correlation  to  reduce  the  variance  of 
the  difference  between  pairs  of  mean  estimates  (Emshoff  and  Sisson, 

1970 ; Fishman,  1973).  The  latter  method  of  using  the  same  sequence  of 
random  numbers  for  each  alternative  is  recommended. 

Continuing  with  our  example,  assume  that  two  blocks  are  generated, 
and  each  alternative  is  replicated  five  times  in  each  block  for  a total 
of  twenty  replications  or  experimental  results.  Let 

xijk  = eneny  casualties  inflicted  by  the  blue  battalion 
during  replication  k for  system  j of  block  i. 

In  our  example,  k = 1,2,..., 5; 

i - 1,2;  and 
i = 1,2. 

The  measure  of  system  1 performance  would  be  the  average_of  sill  observa- 
tions; i.e.,  values  of  X^-j^,  obtained  on  system  1.  Let  X.i.  be  this 
average,  then 


X.i. 
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Throughout  this  report,  average  values  will  be  noted  by  the  symbol  X with 
dots  in  place  of  the  subscripts  averaged  out.  In  this  example,  assume 
that  the  enemy  force  has  at  most  fifty  combatant  weapon  systems  so  no 
value  of  Xjjfc  can  be  greater  than  fifty.  Also  using  the  data  displayed 
in  Table  2.1, 


X.i.  = 15.6 
X.2.  = 18.2  . 


6 


■uw.wt  laiMp— w <*•**  wv** $ tmftpftM*  ^gwr*  -.pr^  •**«• 


Table  2.1 


Red  Casualty  Data 


> 

Rep  1 

Rep  2 

Rep  3 

Rep  4 

Rep  5 

Block  1 

System  1 

15 

16 

17 

18 

15 

System  2 

18 

20 

20 

17 

18 

Block  2 

System  1 

18 

17 

16 

18 

6 

System  2 

18 

23 

17 

21 

iQ 

The  above  system  performance  averages  are  estimates  of  the  true,  but  un- 
known, expected  number  of  red  casualties  for  the  two  systems.  Let 

Ri  = expected  number  of  red  casualties  for  Bystem  1 
Ra  = expected  number  of  red  casualties  for  system  2, 

If  we  let  u be  the  average  value  of  Ri  and  Ra  or 

U = iy  (Ri  + Ra)  » 

then_jx  is  the  overall  average  system  performance.  The  average  of  X.i. 
and  X.2.  is  an  estimator  of  u;  i.e., 

2 (X*i#  + X.2.)  = 16,9  = X. . . 

is  an  estimate  of  u.  Let  Ai  be  the  main  effect  due  to  system  1,  and  As 
be  the  analogue  of  Ai  for  system  2.  These  values  are  determined  by 

Ai  ■ Ri  - u and 
Aa  = R2  - U • 


Note  that  Ai  + Aa  = 0 since 


Ai  + Aa  = Ri  ~u  + R2"U  = Ri"*’  Ra  “ 2u  = 0 . 


sot**# 
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Also  X.i,  - X...  ■ -1.3  is  on  estimate  of  AL  and  X.,».  - X...  - 1.3  is 
the  corresponding  estimate  of  A>.  The  parameters  p,  Aj.,  and  A?  will 
be  in  each  Filter  Model. 

The  values  mentioned  above  account  for  differences  among  systems, 
but  overlook  the  differences  from  block  to  block.  For  example,  the 
block  1 average  performance  over  all  system  replications  is 

Xi..  = 17.4,  and 

x2..  - 16.4 

is  the  block  2 average  performance.  The  above  quantities  are  used  to 
estimate  the  true  block  main  effects,  where 

Bi  = true  block  main  effect,  and 

B2  = analogue  of  Bi  for  block  2. 

The  block  1 expected  average  performance  is  Bi  + p,  and,  similarly  B2  is 
the  block  2 expected  deviation  from  the  overall  true  mean  performance  p. 
Thus,  a very  large  number  of  replications  of  the  block  1 environment 
over  both  systems  would  tend  to  give  values  of  Xi..  close  to  Bi  + p with 
probabilities  approaching  one.  Also,  note  that  Bi  + B2  does  not  necessar- 
ily equal  zero,  since  the  block  main  effects  result  from  random  selection 
of  particular  block  environments  from  the  set  of  all  possible  block  en- 
vironments. This  is  true  although  the  estimators  of  Bi  and  B2  sum  to  zero. 

In  addition  to  the  block  and  system  effects,  an  interaction  is  likely 
to  exist  between  them.  Interaction  is  particularly  likely  in  combat 
systems  since  casualties  tend  to  lead  to  more  casualties,  because  the 
force  Is  weakened,  until  finally  no  more  casualties  can  occur  because 
the  enemy  force  breaks  off  the  engagement  or  becomes  annihilated.  The 
estimators  of  these  interaction  effects  are 

(x^j.  - X,,,)  - (x.^.  " X...)  - (x^..  " X,,.^  = - X.j.  - X^,,  + X... 

for  i = 1,2  and  j = 1,2.  That  is,  the  interaction  effects  estimator  be- 
tween block  i and  system  j is  the  difference  between 

1.  the  deviation  of  block  i and  system  j average  per- 
formance from  the  overall  average  performance;  and 

2.  the  sum  of 

a.  estimated  system  j main  effect,  and 

b.  estimated  block  i main  effect. 

From  the  data  in  Table  2.1,  the  following  estimated  interaction  effects 
are  generated. 
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Estimated  Interaction  Effects 

System  1 System  2 

Block 

1 0.1  -0.1 

2 -0.1  0.1 

In  this  example,  the  interaction  effects  are  smaller  than  the  main 
effects;  however,  this  result  is  not  always  true.  For  example,  a 
block  environment  may  produce  large  average  enemy  casualties  by  one 
system  but  not  the  other.  These  interaction  effects  are  estimators  for 
the  true  interaction  effects  which  are  noted  by 

ABij  = true  interaction  effect  between  block  i and 
system  j. 

Finally,  the  deviations  of  the  individual  experimental  results  from 
their  respective  main  effects  are  defined  as  replication  effects.  These 
replication  effects  are  noted  by 

- replication  effect  for  replication  k for 
system  j of  block  i. 

Collectively,  the  variables  defined  above  lead  to  a model  for  experimental 
results  which  is  employed  by  each  version  of  the  Filter  Model.  That  is, 

xijk  = H + Aj  + Ba  + + eiJk  . 

The  differences  between  each  version  of  the  Filter  Model  occur  due  to 
variations  in  the  assumptions  concerning  the  distributions  of  parameters 
in  the  above  model. 


Filter  Model  Versions 


The  variables  n and  Aj  in  each  version  of  the  Filter  Model  are  fixed 
or  deterministic;  however,  Bi,  ABij,  and  eijk  are  stochastic  or  random 
variables  since  there  exists  a large  number  of  possible  values  for  Bi, 
ABij,  and  ejik*  Because  of  the  manner  in  which  Bi,  ABij,  and  eijk  are 
defined,  their  mean  values  or  expected  values  are  zero.  Three  different 
candidate  Filter  Models  are  defined  in  this  section,  and  each  model  is 
characterized  by  its  assumption  concerning  the  distribution  of  Bi,  ABij, 
and  €ijk.  These  models  are  listed  below  along  with  their  identifying 
acronyms . 

1.  Homogeneous  Variance  and  Independent  Effects  (HOVIE), 

2.  HEterogeneous  Variances  and  Independent  Effects  (HEVIE),  and 

3.  HEterogeneous  Variances  and  Correlated  Effects  (HEVCE). 
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In  addition  to  defining  each  model  in  the  following  sections,  the  esti 
mators  for  model  parameters  are  specified. 

HOVIE  Model 


The  Homogeneous  Variance  and  Independent  Effects  Model  is  identical 
to  the  version  of  the  two-factor  model  commonly  referred  to  as  the  mixed 
model  because  one  factor  is  fixed  and  the  other  random  (Hicks,  1964; 
Graybill,  1961;  and  Winer,  1971)*  In  the  HOVIE  model,  the  random  vari- 
ables Bi,  ABij,  eijfc,  for 

i = 1,2,. . • ,b; 
j = l,2,.»»,a;  and 
k = 1,2,...,  n^ 

are  all  mutually  independent,  where 


b = total  number  of  blocks, 
a = total  number  of  system  alternatives,  and 
nij  = number  of  replications  for  system  j in  block  i. 

The  variances  for  these  random  variables  are 


V(Bi)  = crB2, 

V(ABij)  = crA §,  and 

v(Hjk)  = *e2  • 

Note  that  the  variance  for  each  system  is  identical  to  other  systems. 

The  basic  purposes  of  experimentation  with  simulation  is  to  esti- 
mate the  mean  performance  of  individual  systems  and  differences  between 
the  performance  of  individual  systems.  For  system  j , the  average  per- 
formance is 


X. 


j’ 


nU 

2 

k=l 


and  X.j.  is  an  -unbiased  estimator  of  u + Aj  or  the  system  j mean  perform- 
ance. A preferred  experimental  procedure  would  generate  values  of  X.j. 
with  the  least  possible  cost  for  nominal  variances  of  X.j.  . The  variance 
of  X.j,  is  shown  in  Appendix  D,  equation  D.2,  to  be 


V&j.) 


b 


b 

2 

i=l 
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n 


ij 


(D.2) 
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Note  that  V(X.j.)  is  inversely  proportional  to  b,  and  one  term  contain- 
ing the  variance  of  the  replication  effects  is  reduced  by  increasing 
the  number  of  replications.  If  the  cost  of  adding  replications  is  not 
excessive  and  the  value  of  oe2  is  significantly  larger  than  o^s  + erg2, 
then  multiple  replications  should  be  made  with  each  block.  On  the  other 
hand,  large  values  of  c rA2  + o^f  would  indicate  that  only  one  replication 
of  each  block  should  be  made* 


To  apply  equation  D.2,  estimates  of  the  parameters  Og2  + o^g  and  o-e2 
must  be  generated  from  available  data.  These  estimators  are  noted  as 
Cg2  + °AB  811,1  °e2»  811(1  81,8 


b a 
S S S 
i=l  J=1  I<=1 


niJ  / / r b 


(D.  6) 


b 

;2+?A  = s (5L..  -x...)  / (b-1) 

D AB  1=1 

+ S S (X...  -X  ..  -X.J.  + X. . . )2/[(a— l)(b— 1)] 
i=i  J=1  3 


where 


am  z l -i- 

® j=l  i=l  “ij 


a2b 


(D.7) 
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Xjt  • = “ S Xjji  i and 

1 b - 

X. . . - T"  2 X...  • 

b i=i  1 


These  estimators  are  unbiased  which  me^uis  that  the  expected  value  of  the 
estimator  is  equal  to  the  parameter  value  estimated;  however,  a practical 


problem  may  occur  in  implementing  the  estimator  og2  + o^g,  The  quantities 
og2  and  cta!  are  variances  which  implies  that  they  are  never  negative,  but, 
the  statistic  og2  + may  assume  negative  values  by  chance.  When  this 
occurs,  the  estimator  is  known  to  be  incorrect;  thus,  the  estimate  is 
certainly  improved  by 

if  og2  + 0^1  < 0, 

set  oB2  + aA | = 0. 

The  assumption  is  made  that  the  true  but  unknown  value  of  og2  + cr^g2  is 
probably  a small  positive  number  when  og2  + 8^g2  is  less  than  zero,  so  a 
zero  value  for  eg2  + o^g2  is  fairly  close  to  the  actual  value.  However, 
this  correction  for  negative  values  changes  the  statistic  to  a biased 
statistic  since  the  resulting  value  is  either  equal  to  or  greater  than 
the  unbiased  statistic. 

Another  important  estimator  is  the  difference  between  two  system 
performances,  particularly  when  simulation  experiments  are  being  con- 
ducted in  order  to  filter  out  weaker  or  less  desirable  candidate  systems. 
The  estimator  for  the  difference  between  system  j and  t's  performance  is 


The  efficiency  with  which  filtering  occurs  is  directly  related  to  the 
magnitude  of  the  variance  of  this  difference.  Since  all  random  variables 
in  the  HOVIE  Model  are  assumed  to  be  mutually  independent,  then  the  var- 
iance of  the  difference  is  simply 

V(X.j.  - X.t.)  = V(X.j.)  + V(X.t.)  . 

Fran  a statistical  viewpoint,  then  the  filtering  process  is  more  in- 
efficient than  estimating  the  absolute  magnitude  of  individual  systems. 
The_analyst  should  be  aware  that  a ranking  procedure  based  on  the  values 
of  X.j.;  j = l,2,»»»,a;  is  actually  a number  of  pairwise  comparisons  or 
differences. 

HEVIE  Model 


The  Heterogeneous  Variances  and  Independent  Effects  Model  extends 
the  HOVIE  Model  to  account  for  the  systems  with  unequal  variances.  Other 
analyses  of  DTNCCM  output  data  clearly  indicate  that  variances  of  the 
same  performance  measure  are  unequal  when  the  system  is  changed.  In  this 
model,  the  variances  of  Bi  and  AB^  are  still  Og2  and  o^g2 , respectively, 
but  the  variance  of  the  replication  variation  is  now 

V(eijk)  = <rej2  , j = l,2,»»»,a  . 

The  variance  of  the  system  average  performance,  X, ..  is  derived  in 
Appendix  C and  is  J 
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(C.1) 


which  ie  identical  with  the  corresponding  expression  for  the  HOVIE  Model 
except  oe2  is  replaced  with  aej2.  However,  in  actual  use,  the  allowance 
of  different  values  for  ue2  for  each  system  means  the  least  cost  test  plan 
will  specify  different  numbers  of  replications  for  each  system.  For  the 
HOVIE  Model,  the  system  test  plans  will  all  be  identical  because  the  vari- 
ances are  identical. 

To  use  equation  C,1  estimators  for  aej2  and  erg2  + o^g2  are  required, 
and  the  unbiased  estimator  derived  in  Appendix  C for  is 


r b ”«  , i 

/ 

* b 

2 2 <XIjk-X  )2 

1=1  k=l  1 J 

m m 

/ 

2 (nji  - b) 
1=1 

(C.5) 


Note  that  the  estimator  for  c rej2  is  the  same  as  the  homogeneous  estimator 
except  that  the  data  are  partitioned  and  only  results  for  a given  value 
of  j are  used  to  estimate  ,2 , Thus,  the  gain  in  realism  with  this 
change  is  at  the  expense  of  efficiency  in  estimating  values,  Hie  esti- 
mator for  crBa  + Ojy j derived  in  Appendix  C is 


2 (jq..  -X...)2/  <b-  !) 

1-1 


+ 2 s (Xir  - Xj..  - X + X..)2/l(a-i)(b-i)] 
1=1  j-1  1 


s ? 2 s JL_. 

aZb  j=i  ej  1=1 


(C.  14) 


If  aB2  + < 0,  then  set  0£2  + to  zero  for  the  same  reasons  as 

used  for  the  HOVIE  Model, 


Also,  since  the  randan  variables  in  the  HEVIE  Model  are  assumed  to 
be  independent,  the  variance  of  the  difference  between  two  system  perform- 
ance averages  is 
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v(x«j.  - x,t.)  = v(x.j.)  + v(x,t.)  . 

HEVCE  Model 

The  Heterogeneous  Variance  and  Correlated  Effects  Model  includes  the 
heterogeneous  variances  of  the  HEVIE  Model,  but  alBO  considers  potenti- 
ally important  correlations  in  the  results.  The  significance  of  these 
correlations  are  explained  by  referring  to  the  example  given  earlier  in 
this  chapter. 

The  influence  of  correlations  starts  after  a block  environment  (im- 
plying a value  of  Bj_  for  that  block)  is  obtained.  Assume,  for  example, 
that  a particular  block  environment  tends  to  produce  more  casualties 
than  the  overall  mean  value  of  n (estimated  by  X,,,  = 16.9).  If  so, 
then  B^>0  and,  for  example,  could  be  2,0,  Given  this  block  environment 
would  the  values  of  AB^j;  j = l,2,.».,a;  tend  to  be  greater  than  zero  or 
less  them  zero  on  the  average?  When  the  performance  measure  is  casual- 
ties in  many  systems  the  value  of  the  block  environment  is  likely  to 
affect  the  interaction  term  so  that  B^  > 0 tends  to  produce  larger  values 
of  ABij,  and  Bi  < 0 tends  to  reduce  the  value  of  ABij.  For  example,  the 
block  environment  increasing  block  casualties  by  2,0  out  of  a total 
possible  of  50  may  increase  the  interaction  values  by  25$.  The  covari- 
ance* between  Bi  and  AB-y  indicates  the  degree  of  this  relationship  and 
°BAB  = covariance  between  B^  and  ABjj . Note  that  this  covariance  value 
is  assumed  to  be  the  same  for  all  systems. 

Since  B and  each  one  of  the  interaction  terms,  ABij ; j = l,2,**»,a; 
are  correlated,  then  the  individual  AB^j  values  within  a block  are  likely 
to  be  correlated.  The  covariance*  between  two  values  of  ABij  for  differ- 
ent systems  of  the  same  block  is 

°ABAB  = covar^ance  between  ABj_j  and  AB^ 

where  t ^ J. 

In  addition  to  correlation  among  the  block  and  interaction  effects, 
these  effects  may  be  correlated  with  the  replication  effects,  eijk  . 

To  illustrate  this  point,  assume  the  following  values  for  the  true  param- 
eters 

li  = l6  Aa  = -1 

Bi  = 2 ABn  = 0.3 

Ax  = 1 AB12  *»  -0,3  ) 

where  the  system  performance  meaeure  is  in  enemy  casualties  out  of  a 
total  enemy  force  of  50  weapons.  If  the  mean  enemy  casualties  produced 


*The  covariance  between  two  zero  mean  random  variables  is  defined  as  the 

expected  value  of  their  product. 
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by  this  block  environment  for  system  1 is  19*5,  then  correlations  are 
present  among  the  positive  block  effect  of  Bj.  - 2,  the  interaction  effect 
of  ABii  ,3»  nnd  the  replication  variations  Moreover,  this  corre- 

lation produces  a net  effect  of  ,2  casualties  since  independent  zero  mean 
values  of  would  result  in  a mean  number  of  casualties  for  this  block 
environment  of  19.3.  Note  that  the  overall  mean  for  each  value  of 
continues  to  be  zero,  although  the  expected  value  of  given  values  of 
ABi-j  and  B^  is  probably  not  zero.  Covariances  resulting  from  these  corre- 
lations are  defined  below  for  the  HEVCE  Model. 

°Bej  = covariance  between  the  block  i main  effect  and  a 

replication  effect  for  system  j during  the  ith  block, 

aABe1  = covartance  between  the  interaction  main  effect  for 
system  j and  a replication  effect  for  the  same 
system  during  the  same  block, 

°ABej  = covariance  between  the  interaction  main  effect  for 

system  t and  a replication  effect  for  system  j during 
the  same  block, 

for  j = l,2,««»,a;  t = l,2,*.«,a;  and  t j.  Note  that  the  covariances 
defined  above  have  unique  values  assigned  for  the  system  producing  the 
replication  effect,  and  these  unique  values  are  assigned  because  the 
replication  effects  have  heterogeneous  variances. 

Since  the  replication  effects  are  correlated  with  the  system-block 
interaction  effects  and  the  block  effects,  the  replication  effects  are 
likely  to  be  autocorrelated.  Thus,  there  will  be  correlations  among 
different  replication  effects  of  the  same  system  as  well  as  correlations 
between  replication  effects  of  different  systems.  The  covariances  for 
these  replication  effects  are  defined  below, 

o-gj  = covariance  between  two  different  replication  effects 
of  system  j; 

°ejt  = covariance  between  the  replication  effects  on  the  same 
replication  of  two  different  systems,  i.e.,  j and  t; 

= covariance  between  replication  effects  on  different 
replications  of  different  systems,  i.e,,  j and  t. 

Note  that  for  replication  effects  of  different  systems  a distinction  is 
made  as  to  whether  the  same  replication  is  used  or  a different  replica- 
tion is  involved.  This  distinction  is  made  because  identical  random 
number  sequences  may  be  used  for  each  system  during  the  same  replication. 
By  defining  the  covariances  in  this  way,  the  positive  correlations  are 
considered  among  results  for  identical  replications  of  different  systems, 

A general  relationship  in  the  HEVCE  Model  applies  to  correlations 
and  covariances  among  variables  in  different  blocks.  All  random  variables 
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in  different  blocks  are  considered  to  be  independent;  thus,  their  co- 
variances  are  zero. 

Expressions  for  important  variances  and  estimators  of  model  param- 
eters are  derived  in  Appendix  A for  the  HEVCE  Model,  The  variance  of 
the  average  performance  for  system  j is  given  by  equation 


V(X.j.)  = 


+ 2<r 


BAB  + 2aBej  + ^ABej 


(A.  42) 


where 


bj  = number  of  blocks  having  replications  for  system 
i.e,,  number  of  blocks  where  ny  > 0, 


Parameter  estimation  for  applying  equation  A, 42  is  simplified  consider- 
ably by  estimating  the  following  quantities  in  aggregate  form  rather  than 
their  individual  components: 


1. 


system  j block  variance 

°B  + "AB  + ?ffBAB  + ^Bej  + ^ABej 


and 


2. 


system  j replication  variance 


Both  parameter  values  defined  above  are  heterogeneous  in  the  sense  that 
each  system  has  unique  values. 

An  unbiased  estimator  for  Vy  is  identical,  with  the  estimator  for 
o in  the  HEVIE  Model  defined  earlier.  The  estimator  for  Vr1  is 
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However,  the  estimator  for  Vbj  is  not  equal  to  the  estimator  for 
erg'''  + apj^  in  either  the  HOMIE  or  HEVIE  Models,  If  it  were,  the 
models  would  be  equivalent  from  an  operational  viewpoint.  Hie  esti- 
mator for  Vfcj,  derived  in  Appendix  A,  is 


bj 


<V1)(  f V 

nij>0 


l nij(V 

nij>0 


<v 1 > 


Vrj 


bJ(bJ-2)/(  Sn,j)+  -(  S -) 

• njj>o  n^>o  • 

(A.  52) 


If  VbJ  < 0,  set  = 0. 

Besides  using  a different  statistic,  the  estimator  for  Vbj  only  uses 
data  observed  for  system  j. 


The  principal  application  for  the  Filter  Model  is  to  compare  system 
performances  efficiently;  thus,  the  variance  of  the  difference  in  two 
system  averages  performances  is  important.  The  expression  for  this 
variance  is 


V<V*V>  ■ iLKB^ABcl* 


2c . . + (r 1 . + tr  * . - 2c - 2c ' 


at 


ABej  ABet  ej  et  A BAB  ABej 


‘ *ABet  ‘ 2%jt) 


1 _ 1 1 1 

+ — T V , ? — + 7~7  v . s — 

b,t  rj  i,  n.  b 2 rt  { n 

n1j>0  nij’>o  lt 


2 ^ ejt  ~ ^ ejt^ 
b 2 


i,  njt) 

nij,nlt>0 


(A.  63) 
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where 


b,t  = number  of  blocks  having  replicatins  for  both  systems 
j and  t,  and  the  assumption  Is  made  that 


h3  ~ h t 3 bjt  • 

Also,  estimation  is  faciltated  by  aggregating  parameters  so  that 


2<r  “ _ + 2<r  A + 2<r  + it  * + a- ' _ ?<r 

ABej  ABet  ej  0 et  **  ABAB 


AB 


2 <r 


ABej 


V' 

rjt 


2<r 


ABet 


2 <r 


ejt 


, and 


The  potential  significance  of  increasing  correlation  to  reduce  the  vari- 
ance of  differences  in  average  performance  by  using  identical  random 
number  sequences  is  apparent  from  equation  A.63.  The  identical  random 
number  sequences  should  increase  the  values  of  V' ..  and  positive  values 
of  reduce  the  V(X.j.  - X.t. ).  Unbiased  estimators  for  and 

are  given  in  Appendix  A by  equations  A, 69  and  A.67,  respectively. 


Least  Cost  Experimental  Design 

An  algorithm  is  presented  in  Appendix  B for  calculating  the  least- 
cost  experimental  design  to  reduce  the  estimated  variance  of  each  average 
system  performance  below  a specified  upper  limit,  viz,,  Vs.  Since  the 
estimated  replication  and  block  variances  may  be  different  for  each  system, 
then  the  desired  experimental  design  may  specify  different  replication 
numbers  for  each  system.  The  algorithm  can  be  initialized  with  inputs 
specifying  that  some  experiments  may  already  have  been  performed;  thus, 
the  algorithm  must  determine: 

1.  additional  replications  that  should  be  observed  for 
each  system  in  each  block  previously  generated; 

2.  additional  blocks  that  should  be  generated  and  the 
replication  numbers  for  each  new  block  for  each 
system. 

Two  costs  are  considered  by  the  algorithm;  i.e., 

Ojj  = cost  to  generate  a new  block,  and 

Cj.  = cost  to  perform  a single  replication  for  one 
system  after  the  block  has  been  generated. 
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An  Interesting  aspect  of  the  problem  to  find  the  leasts  cost  experimented 
design  is  that  the  cost  of  generating  a new  block  benefits  all  systems , 
whereas  the  cost  of  an  individual  replication  only  contributes  to  reduc- 
ing the  variance  for  one  system. 
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CHAPTER  3 

EXPERIMENTAL  RESULTS 
Introduction 


A set  of  DYNCOM  experiments  were  generated  in  order  to  indicate  the 
potential  for  the  Filter  Model  and  test  the  concepts  presented  in  Chapter 
2.  The  resulting  experimental  data  are  analyzed  in  this  chapter  in  order 
to  address  the  following  issues: 

1.  Which  Filter  Model  is  the  most  valid  and  useful  representation 
of  DYNCOM  results? 

2.  Does  the  Filter  concept  of  simulation  experimentation  permit 
significant  reductions  in  experimental  cost  to  achieve  a 
specified  variance  of  system  average  performance? 

3.  Does  the  Filter  concept  of  simulation  experimentation  permit 
significant  reductions  in  experimental  cost  to  achieve  a 
specified  variance  of  the  difference  between  the  average 
performance  of  two  systems? 

U.  Can  the  Filter  Model  be  applied  to  screen  candidates  system 
alternatives  using  a low  cost  set  of  DYNCOM  runs? 

A total  of  forty-six  DYNCOM  simulated  battles  were  conducted  in 
June  1972  to  answer  the  questions  posed  above.  The  forces  involved 
consisted  of  four  blue  Armored  Personnel  Carrier  (APC)  Weapons  in 
defensive  positions,  and  thirty -one  attacking  red  tanks.  The  red  tanks 
were  equipped  with  conventional  main  guns , and  the  defending  blue  APC ' s 
were  armed  with  an  anti-tank  missile.  The  red  tanks  were  organized 
into  three  maneuver  units  of  approximately  company  size  consisting  of 
eleven,  ten,  and  ten  tanks. 

Two  system  tactical  alternatives  were  simulated  to  represent 
differences  in  system  effects.  System  one  restricted  the  attacking 
reds  from  opening  fire  until  they  closed  within  1200  meters  of  the 
blues,  and  system  two  represented  a red  opening  fire  range  of  800  meters. 
Because  of  these  opening  fire  ranges,  a block  environment  had  to  occur 
before  a red  would  open  fire  when  employing  the  1200  meter  tactic.  Since 
the  blues  could  open  fire  earlier  than  the  reds,  firing  and  casualties 
could  and  did  occur  before  the  block  environment. 

Experiments  with  four  blocks  were  conducted.  The  block  environment 
was  generated  by  simulating  a complete  battle  and  recording  the  complete 
DYNCOM  data  set  and  status  variables  at  the  time  of  the  block  environ- 
ment during  the  course  of  the  run.  Actually,  the  block  environments 
were  selected  after  the  fact  because  the  battle  generating  the  block 
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environment  would  output  a data  set  and  status  variables  periodically 
during  the  course  of  the  battle.  Because  of  this  preselected  period, 
the  block  environments  could  have  been  selected  more  efficiently  by 
dynamically  determining  the  first  red  firing  event  under  the  1200  meter 
opening  fire  tactic  and  then  recording  the  data  set  and  status  variables 
just  prior  to  this  first  red  firing  event. 

After  obtaining  a block  environment  by  the  procedure  described 
above,  then  the  block  was  replicated  for  each  system.  To  increase  the 
correlation  among  systems  within  each  block,  a common  random  number 
sequence  was  employed  for  each  system  on  a given  replication.  Each 
block  consisted  of  five  replications  of  system  one  and  four  of  system 
two. 


Thirty-six  of  the  simulated  battles  were  conducted  to  replicate 
the  four  blocks  described  above,  and  ten  additional  full-length  battles 
were  simulated  using  system  one  to  serve  as  a basis  for  comparison. 

That  is,  the  variance  of  an  individual  battle  could  be  estimated  by  the 
Filter  Models  from  blocked  data,  and  this  estimate  could  be  compared 
with  the  corresponding  estimated  variance  using  the  ten  independent  battle 
observations. 

Four  performance  measures  that  have  been  employed  by  MICOM  in 
actual  studies  are  used  in  the  analyses  described  in  this  chapter. 

These  system  performance  measures  are: 

1.  red  casualties 

2.  rounds  fired  by  red  weapons  at  blue  weapons 

3.  blue  exposure  time,  and 

4.  first  engagement  range. 

More  explicit  definitions  employed  for  these  measures  as  they  were 
applied  in  this  study  appear  below.  A casualty  must  include  loss  of 
firepower  capability,  so  a complete  kill,  a mobility  and  firepower  kill, 
and  a firepower-only  kill  would  be  counted  as  a casualty.  On  the  other 
hand,  mobility-only  kills  would  not  be  included.  The  rounds-fired 
performance  measure  was  the  total  rounds  fired  by  the  red  tank  main 
gun.  Blue  exposure  time  consists  of  the  cumulative  time  that  an  indi- 
vidual blue  weapon  had  been  acquired  by  red  weapons.  For  example,  if 
blue  1 had  been  acquired  by  red  1 for  50  seconds  and  red  2 for  75 
seconds,  then  the  exposure  time  for  blue  1 is  125  seconds.  An  acquisi- 
tion implies  sufficient  knowledge  to  concentrate  fire  upon  the  acquired 
weapon,  and  this  acquisition  can  be  accomplished  either  by  direct  visual 
contact  or  by  pinpointing.  The  blue  exposure  time  performance  measure 
was  computed  for  an  individual  battle  by  averaging  the  exposure  time 
for  each  blue  weapon.  The  first  engagement  range  is  the  range  at  which 
red  weapons  open  fire  at  the  blue  weapons.  This  range  was  computed  for 
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each  blue  weapon  by  determining  the  maximum  range  that  any  red  weapon 
fired  at  a particular  blue.  In  the  event  no  reds  fired  at  a particular 
blue,  the  minimum  range  that  the  blue  fired  at  any  red  was  taken  as  the 
red  opening  fire  range  so  long  as  the  red  was  permitted  to  open  fire. 
Once  these  ranges  were  determined  for  each  blue  weapon,  they  were  aver- 
aged to  determine  the  first  engagement  range  for  a red. 

The  system  performance  measures  from  each  replication  of  the  forty- 
six  DYNCOM  replications  are  displayed  in  Tables  3*1  through  3.5. 


TABLE  3.1 


Red  Casualty  Data  from  Blocked  Simulation  Replications 


Replication 

System 

Block  1 

1 

1 

12 

1 

2 

16 

2 

1 

12 

2 

2 

13 

3 

1 

12 

3 

2 

19 

4 

1 

17 

4 

2 

21 

5 

1 

15 

Red  Casualties 


Block  2 

Block  3 

Block  4 

11 

15 

18 

21 

18 

18 

13 

16 

17 

10 

20 

23 

16 

17 

16 

15 

20 

17 

18 

18 

18 

20 

17 

21 

11 

15 

6 
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TABLE  3.2 


Rounds  Fired  at  Blue  Weapons  Data  from  Blocked  Simulation  Replications 


Rounds  Fired  at  Blue 


Replication 

System 

Block  1 

Block  2 

Block  3 

Block  4 

1 

1 

17 

17 

8 

17 

1 

2 

9 

0 

4 

4 

2 

1 

15 

13 

15 

14 

2 

2 

5 

4 

6 

2 

3 

1 

15 

7 

18 

21 

3 

2 

4 

4 

3 

6 

4 

1 

19 

11 

21 

8 

4 

2 

8 

0 

10 

3 

5 

1 

8 

15 

20 

6 

TABLE 

3.3 

Blue  Exposure  Time 

Data  from  Blocked  Simulation  Replications 

Blue  Exposure 

Time  in  Seconds 

Replication 

System 

Block  1 

Block  2 

Block  3 

Block  4 

1 

1 

201.07 

404.77 

369.73 

347.44 

1 

2 

277.88 

52.50 

151.38 

152.30 

2 

1 

325.10 

429.99 

705.33 

327.74 

2 

2 

117.69 

111.13 

216.19 

76.225 

3 

1 

366.08 

186.75 

501.91 

427.94 

3 

2 

72.498 

155.53 

167.96 

122.37 

4 

1 

454.72 

354.03 

536.45 

197.05 

4 

2 

148.07 

127.50 

175.96 

110.17 

5 

1 

610.94 

156.09 

637.37 

203.77 
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TABLE  3.4 

Engagement  Range  Data  from  Blocked  Simulation  Replications 


Engagement  Range  in  Meters 

Replication 

System 

Block  1 

Block  2 Block  3 

Block  4 

1 

1 

1135.5 

1148.0  1046.0 

1100.8 

1 

2 

768.25 

780.3  739.75 

725.5 

2 

1 

1064.8 

1145.3  1151.8 

1124.8 

2 

2 

719-75 

756.5  714.50 

749.25 

3 

1 

1127.8 

1100.0  1134.5 

1078.5 

3 

2 

767.75 

712.0  717.25 

733.5 

4 

1 

1076.3 

1029.0  1156.0 

1016.0 

4 

2 

752.75 

783.8  695.5 

791.00 

5 

1 

899.50 

1157.6  1118.5 

1068.5 

TABLE 

3.5 

System  l Performance  Data  from  Independent 
Full  Length  Simulation  Replications 

Replication 

Red  Rounds 

Casualties  Fired 

Blue  Exposure 
Time  (s) 

Engagement 
Range  (m) 

1 

13 

13 

397.29 

IO89.OO 

2 

16 

15 

509.14 

1144.25 

3 

11 

15 

156.09 

1157.75 

4 

14 

6 

159.71 

1045.50 

5 

16 

5 

218.46 

1018.25 

6 

7 

13 

142.71 

1187.00 

7 

14 

6 

97.01 

1078.75 

8 

13 

12 

351.04 

1123.75 

9 

15 

20 

637.46 

1160.00 

10 

10 

8 

190.42 

1062.50 
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Validity  of  the  Filter  Models 


To  test  the  validity  of  the  Filter  Models,  estimates  of  the  vari- 
ance of  an  individual  simulation  result  were  computed  from  the  blocked 
data  using  each  version  of  the  Filter  Model,  and  these  estimates  are 
compared  with  an  estimate  of  the  corresponding  variance  from  the  ten 
independent  full-length  simulation  replications. 

If 

= performance  measure  for  the  ith  simulation 
replication  from  the  full-length  independent 
simulated  battles, 

then  an  estimate  of  the  variance  of  is 

A 10  / 10  \2 

V(Yi)  = £ £ Yi2  - Z YJ  /90 
9 i=l  \i=l  / 


To  estimate  the  variance  of  Xjjfc  from  the  blocked  data,  the  equations 
for  the  V(X.j.)  are  used  for  the  special  case  of  one  block  having  one 
replication.  Of  course,  estimates  for  each  parameter  were  computed 
using  the  estimators  shown  in  Chapter  2.  That  is,  the  following  esti- 
mating equations  for  V(Xijfc)  were  employed. 


HOVIE  Model 

A 


V(Xijk)  = + ^2ab  + ^e 

A.2  , £2  . A 2 


HE VIE  Model 

A 


V(Xijk)  - -2B  + “ AB  + 


HEVCE  Model 
a . 


■ ''bj  + vr3 


Only  values  for  system  one  could  be  compared  because  the  independent 
full-length  simulation  replications  were  only  performed  for  that  system. 


Tables  3.6  through  3»9  present  comparisons  computed  for  the  vari- 
ance estimate  of  the  three  candidate  Filter  Models.  Although  the  vari- 
ance estimates  from  the  independent  full-length  simulation  replications 
are  used  as  the  standard  for  comparison,  recall  that  there  axe  only  ten 
of  these  independent  observations  for  each  performance  measurg.  To 
simplify  the  notation  in  these  tables  ct2b  + is  noted  as  Vbi,  and 
cPe  as  Vyi  for  the  HOVIE  and  HEVIE  Models. 

Inspection  of  these  tables  reveals  that  the  variance  estimates  from 
the  HEVCE  model  are  superior  to  either  of  the  other  two  models.  Ihe 
estimates  from  the  HEVCE  model  are  closer  to  the  estimates  from  the 
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TABLE  3.6 

Estimates  of  the  Variance  of  Total  Red  Casualties 
for  One  Simulation  Replication 
(1200  Meter  Red  Opening  Fire  Range) 


Independent 

Full-Length 

Replications 

HOVIE 

Model 

HEVIE 

Model 

HEVCE 

Model 

X.i. 

12.9 

14.65 

14.65 

14.65 

A 

V 

-1.73 

-1.79 

-.685 

a 

<> 

11.22 

10.68 

10.68 

v(xivk) 

8.10 

11.22 

10.68 

10.68 

TABLE  3.7 

Estimates  of  the  Variance  of  Total  Rounds  Fired 
at  Blue  for  One  Simulation  Replication 
(1200  Meter  Red  Opening  Fire  Range) 


Independent 

Full-Length 

Replications 

HOVIE 

Model 

HEVIE 

Model 

HEVCE 

Model 

7«i« 

11.3 

14.25 

14.25 

14.25 

A 

Vbl 

-1.744 

-.945 

-1.983 

vrl 

16.52 

24.5 

24.5 

$(Xiik) 

24.01 

16.52 

24.5 

24.5 
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TABLE  3.8 


Ertinatee  of  the  Variance  of  Blua  Exposure  Time 
(1200  Meter  Red  Opening  Fire  Range) 

(s) 

Independent 

Full-Length 

Replications 

HOVIE 

Model 

HEVIE 

Model 

HEVCE 

Model 

285.9 

386.8 

386.8 

386.8 

Vbi 

a 

5614. 

6196. 

10340. 

Vn 

A 

10610. 

16425. 

10425. 

V(Xilk) 

32520. 

16224. 

22620. 

26770. 

Table  3.9 


Estimates  of  »e  Variance  of  Engagement  Bange  (m) 
(1200  Meter  Red  Opening  Fire  Range) 

Independent 

X.i. 

Full-Length 

Replications 

HOVIE 

Model 

HEVIE 

Model 

HEVCE 

Model 

1106.7 

1094.0 

1094.0 

1094.0 

vbi 

4305.2 

4369.8 

87.722 

vn 

3259.2 

3900.2 

3900.2 

vfruk) 

3140.7 

7564.9 

8270.0 

3987.9 
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independent  full-length  replications  than  the  HOVIE  Model  for  each 
performance  measure.  When  comparing  the  HEVCE  and  HEVIE  models,  the 
variance  estimates  are  tied  in  two  cases,  viz.,  red  casualties  and  total 
rounds  fired  at  blue.  For  the  blue  exposure  time  and  engagement  range 
performance  measures,  the  IIEVCE  model  is  clearly  superior.  In  fact,  the 
HEVIE  estimates  the  engagement  range  variance  167$  larger  than  the 
estimate  from  the  independent  full-length  replications  whereas  the  HEVCE 
model  is  only  27 % larger.  Because  the  HEVCE  model  appears  to  generate 
significantly  better  variance  estimates  than  the  other  two  experimental 
models,  the  assumptions  made  in  deriving  the  model  must  be  more  valid. 
Thus,  the  HEVCE  model  will  be  the  only  model  analyzed  in  the  remainder 
of  the  report. 

Although,  the  HEVCE  is  preferred  to  the  other  two  candidates,  the 
larger  question  remains  as  to  whether  the  HEVCE  model  is  sufficiently 
valid  to  represent  DYNCOM  results.  Further  inspection  of  tables  3*6 
through  3.9  indicates  that  the  maximum  error  by  the  HEVCE  model  in 
estimating  the  variance  from  independent  full-length  simulation  repli- 
cations was  32 io.  This  result  corresponds  with  an  F statistic  of  1.32; 
however,  the  performance  measures  are  not  normally  distributed. 
Nevertheless,  an  F statistic  of  1.32  would  support  the  validity  of  the 
HEVCE  model  because  the  F statistic  would  have  to  be  greater  than  3»l8  to 
be  significant  at  the  .05  level.  Moreover,  the  result  of  1.32  is  the 
largest  of  four  admitedly  correlated  variance  ratios.  Thus,  these  re- 
sults certainly  support  the  validity  of  the  HEVCE  model  as  being  re- 
presentative of  DYNCOM  experimental  results. 


Cost  of  Estimating  Mean  System  Performance 

In  this  section,  the  costs  of  estimating  mean  system  performance 
using  the  Filter  Model  concept  of  blocking  are  compared  with  the  use  of 
full-length  independent  simulation  replications.  To  compare  these 
experimentation  costs,  the  number  of  simulated  events  is  used  as  a cost 
measure.  That  is,  the  cost  of  generating  a block  is  specified  by  the 
number  of  events  required  to  produce  a block  environment,  and  the  cost 
of  a single  replication  of  the  block  is  measured  by  the  number  of  events 
subsequent  to  the  block  environment  to  complete  the  simulated  battle. 
Thus,  the  cost  components  used  by  the  least-cost  experimental  design 
algorithm  are: 

Ob  = the  expected  number  of  events  to  generate  a new 
block 

Cr  = the  expected  number  of  events  subsequent  to  the 
block  environment  to  simulate  a battle  for  one 
system  alternative. 

Using  the  results  from  four  blocks  and  a toted,  of  thirty-six  replica- 
tions, these  cost  components  are  estimated  to  be 
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events 


Cb  = 249.25 

Cr  = 558.83  events. 

In  addition,  the  cost  of  an  independent  full-length  simulation  replica- 
tion is 

Cb  Cr  = 808.08  events. 

In  addition,  another  cost  measure  is  the  equivalent  number  of  full- 
length  simulation  replications  which  is  calculated  by  dividing  the  total 
number  of  events  expected  for  an  experimented  plan  by  808.08  events.  If 
the  reader  is  interested  in  a dollar  cost  measure,  the  full-length 
simulation  replications  of  the  scenario  described  in  this  chapter  re- 
quired from  thirty  to  thirty-five  computer  service  minutes  on  an  IBM 
360/65  and  the  current  cost  for  MICOM  of  this  machine  is  $100  per  computer 
service  hour  so  a full-length  simulated  battle  would  cost  from  $50  to 
$60.  In  an  actual  study  the  number  of  alternative  systems  requiring 
consideration  may  be  in  the  hundreds  or  thousands. 

Using  the  cost  measures  given  above  for  Cb  and  Cr,  several  com- 
parisons were  made  to  indicate  the  potential  for  the  Filter  Model  con- 
cept in  reducing  simulation  experiments  costs  when  estimating  mean  system 
performance  for  individual  system  alternatives.  These  comparisons 
assumed  that  the  experiments  would  start  with  two  blocks  in  order  to 
estimate  model  parameters.  The  cost  of  two  blocks  consisting  of  five 
replications  of  system  one  and  four  of  system  two  was  10557  events  or 
13.06  equivalent  full-length  simulation  replications.  Note  that  a 
commonly  used  experimental  plan  for  DYNCOM  is  to  replicate  each  alterna- 
tive ten  times  which  would  cost  6.94  full-length  replications  more  than 
the  two  blocks. 

The  comparisons  consisted  of  co-trasting  the  costs  of  estimating 
mean  system  performance  measures  with  specified  variances  by  blocked 
simulation  experiments  versus  independent  full-length  experiments.  Two 
methods  of  determining  variances  for  comparison  purposes  are  used: 

1,  Equivalent  variances  for  both  methods  of  experimentation. 

2.  Variances  less  than  specified  upper  limits  for  both  methods 
of  experimentation. 

Experimental  costs  for  method  2 as  stated  above  were  determined  using 
the  least-cost  experimental  design  algorithm  presented  in  Appendix  B. 

The  effectiveness  of  this  algorithm  is  related  to  the  accuracy  of  its 
inputs , viz . , the  block  and  replication  variances  for  each  system 
alternative.  To  illustrate  the  effect  of  sample  size  for  these  inputs, 
experimental  designs  were  compared  based  upon  input  variances  calculated 
from  both  two  and  four  blocks  of  data. 
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Cost  Comparisons  for  Achieving  Equivalent  Variances 


Tables  3.10  through  3.13  present  cost  comparisons  for  achieving 
equivalent  variances  of  the  average  system  performance  measures,  viz., 
red  casualties,  rounds  fired  at  blue,  blue  exposure  time  and  engagement 
range.  Entries  in  these  tables  were  calculated  by: 

A A 

1.  estimating  the  block,  V^-j,  and  replication,  Vrj  variances 
using  equations  A. 52  and  A.  47,  respectively,  and  all  four 
blocks  of  data. 

2.  calculating  V(X.j.)  the  estimated  variance  of  the  average 
performance  for  system  j,  by  inserting  the  estimates 

and  Vrj  into  equation  A.  53* 


3.  calculating  the  estimated  variance  of  a full-length  simulation 
replication  of  system  by 

v(x1Jk)  - Vbj  + vrJ 


4.  determining  the  number  nj  of  independent  full-length 
simulation  replications  to  realize  a variance  for  the 
average  system  performance  equal  to  V(X.j.)  by 


nf 


tyx.j.) 


5.  estimating  the  cost  of  independent  full-length  runB  by 
nf  • 808.08  events. 

A 

Any  negative  values  resulting  from  the  computational  equation  for 
are  shown  in  the  tables  although  these  negative  values  are  set  to  zero 
in  Bteps  2 and  3 above.  Note  that  the  Filter  Model  parameters  were 
estimated  using  all  available  data  in  step  1.  This  was  done  to  provide 
an  accurate  picture  of  the  relationship  between  the  test  design  and  esti- 
mated variances.  However,  the  comparison  is  made  with  estimated 
variances  resulting  from  a test  plan  involving  only  two  blocks. 


Hesults  from  the  cost  comparisons  to  achieve  equal  variance  esti- 
mates are  mixed.  The  Kilter  Model  concept  of  blocking  achieved  red 
casualty  variances  in  13.06  equivalent  full-length  runs  that  would  re- 
quire 20  independent  full-length  runs;  thus  blocking  permits  a reduction 
of  35#  in  experimentation  costs  in  this  case.  On  the  other  hand, 
variances  for  rounds  fired,  blue  exposure  time,  and  engagement  range 
would  require  about  the  Bame  costs  for  each  experimentation  method. 

Thus,  the  Filter  Model  concept  of  blocking  is  likely  to  reduce  experi- 
mentation costs  for  some  performance  measures,  viz.,  casualties,  but 
not  all  performance  measures. 
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Table  3.10 


Cost  Comparison  to  Achieve  Equivalent  Variance  of  Average  Red 
Casualties,  Two  Blocks  Versus  Independent  Full-Length  Runs 
(All  Parameters  Estimated  Using  Four  Blocks  of  Data) 


System  1 

System  2 

(OFR  <r  1200  m) 

(OFR  < 800  m) 

A 

vbJ 

-.685 

-.844 

A 

Vrj 

10.68 

11.94 

• 

l>< 

<> 

1.068 

1.492 

v(x1Jk) 

10.68 

H.94 

Cost  of  independent  runs 

8080.1  events  or  8080.1  events  or 

10  runs 

10  runs 

Total  cost  of  independent  runs 

l6l6l  events  or 

20  runs 

Cost  of  blocked  runs 

10559  events  or 

13.06  runs 

Table  3.11 

Cost  Comparison  to  Achieve  Equivalent  Variance  of  Rounds  Fired 
at  Blue,  Two  Blocks  Versus  Independent  Full-Length  Runs 
(All  Parameters  Estimated  Using  Four  Blocks  of  Data) 


System  1 
(OFR  < 1200  m) 

System  2 
(OFR  < 800  m) 

A 

Vbj 

-1.983 

2.656 

-*"D 

u 

<> 

24.5 

5.875 

V(X.j.) 

2.45 

2.063 

v(xiJk) 

24.5 

8.531 

Cost  of  independent  runs 

8080.1  events  or  3343  events  or 

10  runs  4.13  runs 

Total  cost  of  independent  runs 

11423  events  or 

14.13  runs 

Cost  of  blocked  runs 

10557  events  or 

13.06  runs 
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Table  3.12 


Cost  Comparison  to  Achieve  Equivalent  Variance  of  Blue  Exposure 
Time,  Two  Blocks  Versus  Independent  Full-Length  Runs 
(All  Ifcrameters  Estimated  Using  Four  Blocks  of  Data) 


Vbj 

A 

Vrj 

V(X.j.) 

V(Xijk> 

Cost  of  independent  runs 


Total  cost  of  independent  runs 
Cost  of  blocked  runs 


System  1 
(OFR  < 1200  m) 

10340. 

16425. 

6812.5 

26765. 


System  2 
(OFR  < 800  m) 

301.34 

2855.2 
507.57 

3156.3 

5025.4  events  or 
6.22  runs 


3174.9  events  or 
3.93  runs 


8200.3  events  or  10.15  runs 
10557  events  or  13. 06  runs 


Table  3.13 


Cost  Comparison  to  Achieve  Equivalent  Variance  of  Engagement 
Range,  Two  Blocks  Versus  Independent  Full-Length  Runs 
(All  Parameters  Estimated  Using  Four  Blocks  of  Data) 


A 


V(X.j.) 

vOtijk) 

Cost  of  independent  runs 


Total  cost  of  independent  runs 
Cost  of  blocked  runs 


System  1 
(OFR  < 1200  m) 

87.72 

3900.2 

433.88 

3987.9 

7434.6  events  or 
9.20  runs 

11485.2  events  or 


System  2 
(OFR  < 800  m) 

172.89 

697.40 

173.62 

870.29 

4050.6  events  or 
5.01  runs 

14.21  runs 


10557  events  or  13.06  runs 
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Cost  Comparisons  for  Reducing  Variances  Belov  Specified  Upper  Limits 

Tables  3.14  through  3.17  present  the  comparisons  of  cost  to  reduce 
the  variances  of  average  system  performance  below  specified  upper  limits. 
The  situation  assumed  for  these  comparisons  involved  an  initial  set  of 
experiments  and  then  additional  replications  or  blocks  as  needed  for  each 
system  to  realize  the  targeted  upper  limit  on  the  variance  of  average 
system  performance.  This  initial  commitment  to  experiments  on  each 
system  is  assumed  to  be  required  in  order  to  estimate  variances  or  verify 
earlier  estimates.  For  the  blocked  runs  the  initial  commitment  consisted 
of  two  blocks  with  five  replications  of  system  one  and  four  of  system 
two,  and  the  corresponding  initial  commitment  for  the  independent  full- 
length  replications  consisted  of  seven  replications  for  system  one  and 
six  of  system  two.  The  total  cost  in  each  case  of  the  initial  set  of 
experiments  was  approximately  13  equivalent  full-length  runs. 

To  reduce  the  effect  of  sampling  errors,  all  variance  parameters 
used  to  construct  the  comparisons  in  Tables  3.14  through  3.17  were  esti- 
mated using  four  blocks  of  simulation  replications.  This  procedure  is 
consistent  with  the  comparisons  made  in  the  previous  section  to  compare 
costs  with  equivalent  variances. 

The  results  displayed  in  tables  3.14  through  3.17  were  calculated 
by  the  least-cost  experimental  design  algorithm  to  reduce  average  red 
casualty  variances  below  1.00,  average  rounds  fired  variances  below  1.20, 
exposure  time  variances  below  3150,  and  engagement  range  variances  below 
350.  These  results  permit  the  following  observations.  Experimentation 
costs  would  be  decreased  through  the  use  of  blocking  to  estimate  red 
casualties  by  25$,  rounds  fired  by  l6$,  and  engagement  range  by  12$. 
However,  the  costs  appear  to  increase  to  estimate  average  exposure  time 
variance.  Note  that  the  least-cost  algorithm  specifies  that  no  new 
blocks  should  be  generated  to  estimate  average  red  casualties  and 
engagement  range;  however,  four  new  blocks  should  be  generated  to  esti- 
mate average  rounds  fired.  Moreover,  the  algorithm  specifies  additional 
blocks  with  a single  replication  on  each  system  to  estimate  average  blue 
exposure  time.  This  test  plan  is  equivalent  to  independent  full-length 
simulation  replications  which  would  be  less  expensive  in  this  case  than 
replicating  blocks. 

An  overall  evaluation  of  the  Filter  concept  of  blocking  based  upon 
these  results  is  that  the  concept  permits  reduction  in  experimentation 
costs  depending  on  the  performance  measure  considered.  However,  the 
reductions  are  not  dramatic.  As  far  as  the  exposure  time  case  where 
costs  increased  due  to  blocking,  this  result  could  be  avoided  if  pre- 
vious experience  indicated  the  inefficiency  of  blocking  and  simulation 
experiments  were  to  be  set  up  primarily  to  estimate  exposure  time. 
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Table  3.14 


Cost  Comparison  to  Achieve  Average  Red  Casualties  Variance 
Lower  Than  1.00  - Additional  Replications  After  Two 
Blocks  Versus  Independent  Full-Length  Runs 
(All  Parameters  Estimated  Using  Four  Blocks  of  Data) 


System  1 
(OFR  < 1200  m) 

System  2 
(OFR  < 800  i 

Additional  replications 

in  previous  blocks 

1 

2 

Additional  blocks 

0 

0 

Replications  in 

additional  blocks 

0 

0 

Total  cost  of  blocked  runs 

13912  events  or  17.22 

runs 

Cost  of  independent 

8889  events  or 

9697  events 

full-length  runs 

11  runs 

12  runs 

Total  cost  at  independent 

full-length  runs 

I8586  events  or  23  runs 

Table  3.15 

Cost  Comparison  to  Achieve  Average  Rounds  Fired  Variance 
Lower  Than  1.20  - Additional  Replications  After  Two 
Blocks  Versus  Independent  Full-Length  Runs 
(All  Parameters  Estimated  Using  Four  Blocks  of  Data) 

System  1 System  2 

(OFR  < 1200  m)  (0FR  < 800  m) 

Additional  replications 

in  previous  blocks  0 0 

Additional  blocks  4 4 

Replications  in 
additional  blocks 

Total  cost  of  blocked  runs 

Cost  of  independent 
full-length  runs 

Total  cost  of  independent 

full-length  runs  23434  events  or  29  runs 
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3 1 

20495  events  or  25.36  runs 


16970  events  or 
21  runs 


6465  events  or 
8 runs 


Table  3.16 


Cast  Comparison  to  Achieve  Average  Blue  Exposure  Time  Variance 
Lower  Than  3150  - Additional  Replications  After  Two 
Blocks  Versus  Independent  Full-Length  Runs 
(All  Parameters  Estimated  Using  Four  Blocks  of  Data) 


Additional  replications 
in  previous  blocks 

Additional  blocks 

Replications  in 
additional  blocks 

Total  cost  of  blocked  runs 

Cost  of  independent 
full-length  runs 

Total  cost  of  independent 
full-length  runs 


System  1 
(OFR  < 1200  m) 


0 

6 


System  2 
(OFR  < 800  m) 


0 

1 


1 1 
15405.5  events  or  19.06  runs 


7273  events  or 
9 runs 


4848  events  or 
6 runs 


12121  events  or  15  runs 


Table  3.17 


Cost  Comparison  to  Achieve  Average  Engagement  Range  Variance 
Lower  Than  350  - Additional  Replications  After  Two  Blocks 
Versus  Independent  Full-Length  Runs 
(All  Parameters  Estimated  Using  Four  Blocks  of  Data) 


Additional  replications 
in  previous  blocks 


System  1 
(OFR  < 1200  m) 


System  2 
(OFR  < 800  m) 


2 


0 


Additional  blocks 


0 


0 


Replications  in 
additional  blocks 


0 


0 


Total  cost  of  blocked  runs  12792  events  or  15.83  runs 


Cost  of  independent 
full-length  runs 


9697  events  or 
12  runs 


4848  events  or 
6 runs 


Total  cost  of  independent 
full-length  runs 


14545  events  or  18  runs 
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Sensitivity  of  Experimental  Plans  to  Input  Variability 

Ideally,  the  simulation  experimenter  will  have  an  ample  data  base 
to  estimate  parameters  and  plan  blocked  simulation  experiments.  With 
precise  values  of  the  block  variances,  V^j,  and  replication  variances, 

Vr^,  the  DYNCOM  experiments  can  be  designed  so  that  least-cost  experi- 
ments are  always  implemented.  If  independent  full-length  experiments 
are  the  least-cost  strategy,  then  the  least-cost  algorithm  will  specify 
blocks  with  single  replications. 

More  commonly,  the  values  of  Vvj  and  Vr-j  will  have  to  be  estimated 
as  part  of  the  experimental  plan.  The  sensitivity  of  these  parameter 
estimates  and  the  resulting  experimental  plans  to  sample  size  are 
investigated  in  this  section  by  comparing  the  results  obtained  from 
two  blocks  with  results  from  four  blocks  (including  the  former  two 
block  results). 

Table  3*18  presents  the  parameter  estimates  for  both  two  and  four 
blocks.  Recall  that  negative  values  of  will  be  regarded  as  zero 
values  in  deriving  test  plans.  Of  course,  a block  variance  of  zero 
has  special  significance  in  that  it  implies  that  nothing  is  to  be 
gained  by  generating  new  blocks.  The  results  for  two  blocks  show  six 
out  of  eight  cases  having  negative  estimates  for  the  block  variances; 
however,  three  of  these  results  are  changed  when  two  more  blocks  are 
obtained.  One  would  expect  that  the  true  values  of  the  block  variances 
to  be  greater  than  zero,  but  they  are  hopefully  small  to  the  point  of 
being  negligible  when  the  estimator  of  is  negative.  This  rationale 
is  supported  by  two  of  the  three  cases  where  changes  from  negative 
to  positive.  That  is,  for  system- two  exposure  time  changes  from  a 
negative  value  to  30.. 34  where  the  value  of  the  replication  variance  is 
9*5  times  larger  or  2855*0,  and  the  system  2 block  variance  estimate  for 
engagement  range  changed  from  a negative  value  to  172.89  where  the  re- 
plication variance  is  four  times  larger  or  697*40.  A small  block  variance 
relative  to  the  replication  variance  will  result  in  a least-cost  design 
with  a large  number  of  replications  per  block.  A less  desirable  situa- 
tion occurs  in  the  system-one  exposure  time  case  where  a negative  block 
variance  estimate  changes  to  10339  when  four  blocks  are  run,  and  this 
block  variance  is  (>%  as  large  as  the  replication  variance  of  16425. 

The  effects  of  these  variations  in  estimated  values  are  shown  in 
Tables  3*19  and  3.20  where  test  plans  are  compared  based  on  estimate 
from  two  versus  four  blocks  of  data.  Each  test  plan  is  designed  to 
reduce  the  variance  of  average  system  performance  below  the  same  upper 
limits  specified  in  Tables  3*14  through  3.17.  These  upper  limits  are 
noted  as  Vs  in  Tables  3.19  and  3.20.  Two  sources  of  differences  between 
the  test  plans  are  present  in  these  tables: 

a.  change  in  the  overall  magnitude  of  the  variance  of  average 
system  performance 

b.  change  in  relationship  between  the  block  and  replication 
variance. 
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Table  3.18 


Comparison  of  the  Estimates  of  HEVCE  Parameters 
from  Two  and  Four  Blocks  of  Data 

Red  Casualties  Rounds  Fired  at  Blue 


2 Blocks 

4 Blocks 

2 Blocks 

4 Blocks 

System  1 (OFR  < 1200  m) 

Vbi 

-1.480 

-.685 

-.780 

-1.983 

Via 

7.500 

10.675 

16.000 

24.500 

System  2 (OFR  < 800  m) 

A 

Vba 

-4.458 

-.844 

8.750 

2.656 

A 

Vra 

18.958 

11.938 

5.500 

5.875 

Exposure 

! Time 

Engagement  Range 

2 Blocks 

4 Blocks 

2 Blocks 

4 Blocks 

System  1 (OFR  < 1200  m) 

A 

Vb! 

-130.7 

10339. 

331.73 

87.72 

A 

Via 

19600. 

16425. 

5972.7 

3900.2 

System  2 (OFR  < 800  m) 

A 

Vba 

-311.56 

301.34 

-183.21 

172.89 

A 

Vr2 

4836.4 

2855.0 

804.83 

697.40 
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^ Table  3.19 


Comparison  of  Least-Cost  Experimental  Plans  Subsequent  to  Two  Blocks 
(Two  Block  Parameter  Estimates  Versus  Four  Block  Estimates) 


T 

Red  Casualties 

Rounds  Fired  at  Blue 

L 

Vs  = 

1.00 

Vs  = 

1.20 

2 Blocks 

4 Blocks 

2 Blocks 

4 Blocks 

I 

Estimates 

Estimates 

Estimates 

Estimates 

*» 

System  1 (OFR  < 1200  m) 

T «• 

5 . 
; 

Additional  replications 

« * 

in  previous  blocks 

0 

1 

0 

0 

7 * 

1. 

Additional  blocks 

0 

0 

10 

4 

Replications  in 

additional  blocks 

0 

0 

1 

3 

L 

System  2 (OFR  < 800  m) 

i : 

Additional  replications 

i 

• r« 

in  previous  blocks 

6 

2 

0 

0 

jF 

Additional  blocks 

0 

0 

10 

4 

Replications  in 

T * 

additional  blocks 

0 

0 

1 

1 

T " 

• » 

r 


i 


I 
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Table  3.20 


Comparison  of  Least-Cost  Experimental  Plans  Subsequent  to  Two  Blocks 
(Two  Block  Parameter  Estimates  Versus  Four  Block  Estimates) 


Blue  Exposure  Time 
Vs  = 3150 

2 Blocks  4 Blocks 
Estimates  Estimates 


Engagement  Range 


1 8 

2 Blocks 
Estimates 


Va  = 350 


4 Blocks 
Estimates 


System  1 (OFR  < 1200  m) 

Additional  replications 

in  previous  blocks  0 

Additional  blocks  0 


0 

6 


0 

3 


2 

0 


Replications  in 
additional  blocks 

System  2 (OFR  < 800  m) 


Additional  replications 

in  previous  blocks  0 

Additional  blocks  0 


0 

1 


0 

0 


0 

0 


Replications  in 
additional  blocks 
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Note  that  the  major  source  of  differences  between  the  test  plans  is  the 
magnitude  of  the  test  effort  rather  than  the  allocation  of  effort  between 
blocks  and  replications. 


Cost  of  Comparing  Mean  System  Performances 

Probably  the  most  important  application  of  DYNCOM  is  to  compare 
mean  system  performance  in  order  to  determine  which  systems  are  superior 
and  by  what  margins.  Thus,  the  differences  between  average  system  per- 
formances may  be  more  important  to  the  analyst  than  the  absolute  magnitude 
of  average  system  performances.  The  Filter  concept  of  blocked  experiments 
contributes  to  improving  the  efficiency  of  estimating  the  differences 
among  system  performances  by  increasing  the  correlation  among  results  for 
different  systems.  The  use  of  blocks  with  fixed  block  environments  for 
each  replication  and  the  use  of  identical  random  number  sequences  for 
each  alternative  replicated  within  a block  causes  the  desired  correlation. 
Considering  these  correlations,  equation  A. 63  gives  the  variance  of  the 
difference  in  average  performance  between  two  systems.  Thus,  A. 63  can 
be  used  to  estimate  the  degree  of  improvement,  if  any,  in  efficiency 
resulting  from  the  Filter  concept  of  blocking. 

In  this  section,  the  experimental  results  are  analyzed  to  answer 
two  questions: 

1.  are  the  assumptions  leading  to  equation  A. 63  for  determining 
the  variance  of  the  difference  in  average  performance  valid 
for  representing  DYNCOM  results? 

2.  does  the  Filter  concept  of  blocked  simulation  experiments 
permit  savings  in  estimating  the  variance  of  the  difference 
in  average  performances  of  two  systems? 


Validity  of  the  Expression  for  V(X.j.  - X.f) 

To  test  the  validity  of  equation  A. 63,  the  variance  of  the  difference 
between  the  average  performance  of  systems  one  and  two  was  computed 
directly  using  four  replications  of  each  block,  i.e.,  the  fifth  repli- 
cation of  system  one  was  ignored.  For  each  replication,  the  difference 
between  system  one  and  system  two  performance  was  calculated.  That  is, 

Yik  = difference  between  system  one  and  two  performance  for  the 
kth  replication  of  block  i 


Yfk  = Xiik  - Xi2k 

Then  the  values  of  Y^  were  inserted  into  equations  A. 47  and  A. 52  to 
estimate  Vr£  and  V^,  the  replication  and  block  variances  for  differences 
between  system  performance.  The  subscript  d is  used  to  denote  differences. 

A A , 

Then  the  quantities  Vrd  and  Vfcd  are  substituted  into  equation  A. 42  to 
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estimate  the  variance  of  the  difference  between  system  one  and  system 
two  performance.  This  variance  estimate  is  noted  as  V(Y). 
a 

Values  of  V(Y)  are  shown  in  table  3*21  and  compared  with  estimates 
of  V(X.i.  - X.2.)  computed  from  equation  A. 63  using  all  four  blocks  of 

data^with  the  fifth  replication  of  system  one  deleted.  Note  that  V(Y) 
and  V(X. . . - X.2.)  are  identical  for  each  performance  measure  with  the 
exception  of  engagement  range.  In  the  engagement  range  case,  the  value 

A 

of  Vfc  is  estimated^as  -275*14  and  then  set  to  zero  as  specified  in 

equation  A. 52.  If  V^i  retained  the  -275*14  value  and  this  value  was 

used  in  estimating  V(X.i.  - X.p.),  then  \}(X.i.  - X.2.)  would  become  267*14 

a 

and  be  identical  to  V(Y)  for  engagement  range. 

Thus,  the  close  match  among  the  two  alternative  methods  of  calculating 
variances  of  differences  in  average  performance  serves  to  support  the 
validity  of  equation  A. 63.  Although  the  engagement  range  case  suggests 
that  negative  values  of  should  be  retained,  this  modification  is  not 

recommended  because  of  the  possibility  that  estimates  of  V(X.j..  - X.2.) 
may  also  become  negative. 


Table  3*21 

Variance  of  the  Difference  in  Average  Performances  Validity  Check 
(Four  Blocks  With  Four  Replications  Per  System  Per  Block) 


Red 

Casualties 

l>^ 

<> 

*7904 

v(x.t.  - X.,.; 

.7904 

v(xM  .)  + v(x.,.) 

1*5846 

F 

.4988 

Rounds  Fired 
at  Blue 

Exposure 

Time 

Engagement 

Range 

1.1745 

1371*7 

267.14 

1.1745 

1371.7 

335-93 

2.3125 

2615.2 

230.59 

.5079 

.5245 

1.457 
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Cost  of  Estimating  Differences  in  Average  System  Performance 

Besides  showing  estimates  of  the  variance  of  the  difference  in 
average  system  performances,  Table  3*21  computes  estimates  of  these 
variances  under  the  assumption  that  observations  of  system  one  and 
system  two  are  independent.  Under  the  assumption  of  independence,  the 
variance  of  the  difference  in  average  performance  is  estimated  by: 

V(X.i.  - X ,2 . ) = V(Xn.)  + V(X.2.) 

With  correlation,  the  V(X.i.  - X.2.)  becomes  much  smaller  as  shown  in 
Table  3.21.  The  values  of  F shown  in  the  table  are  the  ratio  between 

A — — 

the  estimates  of  V(X.i.  - X.2.)  and  the  estimate  under  the  assumption  of 
independence.  Most  of  the  F values  are  approximately  .5  implying  a 50$ 
reduction  in  variance  by  the  Filter  Model  concept  of  blocking.  The  only 
exception  is  engagement  range,  where  F is  approximately  1.5  suggesting 
larger  variances  of  differences  than  independent  sampling. 

A better  comparison  between  the  variances  of  differences  in  per- 
formance under  independent  and  blocked  experiments  is  shown  in  Table  3.22. 
All  of  the  blocked  data  including  the  fifth  observation  on  system  one 
are  used  in  the  variance  estimates.  Again,  the  effect  of  blocking  permits 
a significant  reduction  in  variance  with  exception  of  engagement  range. 

For  rounds  fired,  the  variance  under  blocking  is  less  than  40$  of  its 
values  under  independent  sampling.  Differences  in  red  casualties  and 
exposure  time  have  variances  under  blocking  of  50$  of  corresponding  values 
under  independent  sampling.  The  engagement  range  case  must  have  some 
negative  correlation,  i.e.,  a high  engagement  range  in  a battle  where 
the  reds  can  open  fire  at  1200  metters  would  tend  to  produce  a low  en- 
gagement range  when  the  800  meter  opening  fire  tactic  is  played.  However, 
if  the  analyst  is  primarily  interested  in  casualties,  rounds  fired,  and 
exposure  time;  the  Filter  Model  concept  of  blocking  will  produce  an 
estimated  reduction  in  variance  of  50$  for  a specified  experimentation 
cost. 


Table  3.22 

Variance  of  the  Difference  in  Average  Performance 
(Four  Blocks  With  Five  Replications  of  System  One  and  Four  of  System  Two) 


Red 

Casualties 

Rounds  Fired 
at  Blue 

Exposure 

Time 

Engagement 

Range 

• 

CM 

l>< 

<> 

+ 

• 

ix 

<> 

1.2798 

2.2563 

3660.0 

303.75 

v(x.t . - x.3.) 

.6403 

.8797 

1874.7 

436.98 

F 

.5003 

.3899 

.5122 

1.439 
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Screening  Candidate  Alternatives 


A primary  objective  for  the  Filter  Model  is  to  screen  or  filter 
candidate  systems  with  low-cost  replications  of  DYNCOM  in  order  to  elim- 
inate weaker  systems.  In  this  section,  a single  block  with  five  re- 
plications on  each  system  is  evaluated  with  respect  to  its  efficiency 
in  filtering  candidate  systems.  Based  on  the  experiments  evaluated  in 
this  report,  the  cost  for  two  systems  is  7.22  equivalent  full-length 
replications.  To  evaluate  this  filter,  the  variance  of  the  difference 
in  average  casualties  is  computed  for  each  performance  measure,  and  the 
cost  in  independent  full-length  replications  to  achieve  the  same  variance 
is  computed. 

To  estimate  the  variance  of  the  difference  in  average  casualties, 

equation  A.63  becomes  V(X.lt  - X.,.)  = V&2  + (V^  + V*,)/!?  - 2V^2/5 
for  this  case  of  one  block  with  five  replications  of  each  system.  Also, 

negative  values  of  V^?  are  set  to  zero  in  order  to  avoid  the  possibility 

of  V(Xn.  - X.2.)  becoming  negative.  Values  of  VCX^.  - X.2.)  are 
presented  in  Table  3-23  for  each  performance  measure.  The  cost  in  equi- 
valent independent  full-length  replications  to  match  V(X.L.  - X.2.)  is 
determined  by 

1.  computing  the  variance  of  the  difference  for  one  full-length 
replication  on  each  alternative  system  by 

v(xllk)  + v(xl2k) 

2.  computing  the  cost  to  match  the  single  block  filter  by 

V(Xllk)  + $(X1?k) 

2 A _ _ . 

V(Xn  . - X.,.) 

equivalent  full-length  replications. 

The  calculation  procedure  shown  above  assumes  that  each  system  will  have 
the  same  number  of  replications  in  both  the  Filter  and  independent  full- 
length  replications. 

The  cost  comparisons  in  Table  3*23  show  a very  large  cost  reduction 
for  the  Filter  in  comparing  red  casualties  and  rounds  fired.  These 
performance  measures  would  require  23.08  full-length  runs  for  casualties 
and  20.48  runs  for  rounds  fired  to  match  the  Filter  variance  obtained 
with  only  7.22  runs.  In  other  words,  the  single  block  Filter  is  about 
three  times  more  efficient  than  independent  full-length  replications  for 
casualties  and  rounds  fired.  However,  the  exposure  time  and  engagement 
range  performance  measures  show  about  the  same  cost  for  the  Filter  and 
independent  full-length  replications. 
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Table  3.23 


I 

1 

Efficiency  of  a Single  Block  Filter  With  Five  Replications 


Red 

Casualties 

Rounds  Fired 

Exposure 

Time 

Engagement 

Range 

A 

Vdl3 

5.1229 

-.2938 

2306. 

3737.2 

U 

<> 

10.675 

24.50 

16426. 

3900.2 

B 

<> 

11.938 

5.875 

2855. 

697.4 

19.203 

7.125 

-2984.6 

7359.3 

v(x.t . - x.3.) 

1.964 

3.225 

7356.0 

1713.05 

j 

V(xllk) 

10.675 

24.50 

26765. 

3987.9 

^(xi2k) 

11.938 

8.53 

3156.5 

870.29 

Cost  of  independent 
replications 

23.03  runs 

20.48  runs 

8.14  runs 

5.67  runs 

Cost  of  filter 
replications 

7.22  runs 

7.22  runs 

7.22  runs 

7.22  runs 

IE 
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CHAPTER  4 

SUMMARY  AND  CONCLUSIONS 


In  Chapter  2 several  experimental  design  models  were  hypothesized 
for  the  purpose  of  representing  the  results  of  blocked  DYNCOM  runs.  The 
objective  for  these  models  is  to  investigate  the  economics  of  operating 
DYNCOM  to  generate  a battle  situation  or  block  environment,  and  then 
replicating  the  remainder  of  the  battle  for  each  alternative.  These 
blocked  replications  would  be  useful  if  the  variance  in  average  system 
performance  or  the  variance  of  the  difference  in  average  system  per- 
formance for  two  systems  were  less  for  a fixed  computer  expense,  A 
primary  objective  would  be  to  use  one  or  more  inexpensive  blocks  to 
filter  candidates  to  eliminate  weaker  alternatives.  This  Filter  is  re- 
quired to  be  inexpensive  and  have  small  variances  for  the  difference 
in  average  performances  for  two  systems. 

Several  models  were  proposed  because  the  assumptions  inherent  in 
experimental  models  that  are  available  in  the  literature  appear  inappro- 
priate for  a combat  model  like  DYNCOM.  The  primary  extensions  made  to 
derive  the  Filter  Experimental  Model  involve  correlations  among  block, 
interaction,  and  replication  effects  and  heterogeneous  variances. 

A set  of  DYNCOM  experiments  were  conducted  at  MICOM  to  evaluate 
these  Filter  Models  and  test  their  efficiency.  These  experiments  in- 
volved two  system  tactical  alternatives  where  the  opening  fire  range 
for  the  attacking  red  force  was  set  at  1200  meters  for  system  one  and 
800  meters  for  system  two.  Four  performance  measures  were  analyzed, 
viz.,  red  casualties , rounds  fired  at  blue  weapons,  blue  exposure  time, 
and  first  engagement  range.  An  important  characteristic  of  the  experi- 
ment was  that  a common  random  number  sequence  was  used  for  each  repli- 
cation in  a block  on  each  system  in  order  to  increase  the  correlation 
among  the  results  for  the  two  systems.  By  increasing  correlation,  the 
variance  of  the  difference  in  average  performance  is  reduced.  The  re- 
sults of  the  analysis  of  the  experiment  are  presented  in  Chapter  3»  Of 
course,  these  results  should  be  interpreted  as  illustrating  the  potential 
for  the  Filter  Model  concept  of  DYNCOM  experimentation,  and  results 
for  other  systems  or  battlefield  environments  maybe  different. 

The  overall  conclusions  drawn  from  these  experiments  are  that  the 
Filter  Experimental  Model  is  a valid  representation  of  DYNCOM  experi- 
mental results,  and  that  significant  economies  can  be  realized  by 
using  DYNCOM  as  a filter  depending  upon  the  analysts  choice  of  perfor- 
mance measures  for  ranking  the  system  alternatives.  Excellent  results 
were  obtained  in  reducing  simulation  costs  if  the  performance  measures 
of  interest  were  red  casualties  or  rounds  fired.  For  example,  a DYNCOM 
filter  using  one  block  and  five  replications  per  alternative  obtained 
estimated  variances  for  the  difference  in  average  red  casualties  or 
rounds  fired  in  approximately  one-third  the  cost  of  independent  full- 
length  runs.  Results  for  blue  exposure  time  and  engagement  range  showed 
no  reduction  in  simulation  expense. 
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When  using  more  than  one  block  to  compare  system  performances  by 
computing  the  differences  in  system  performance,  the  Filter  concept  of 
blocking  continues  to  offer  the  simulation  experimenter  more  for  a fixed 
level  of  effort.  Results  obtained  from  four  blocks  indicate  that  the 
Filter  experimental  procedure  gives  a variance  one-half  of  the  variance 
expected  on  the  basis  of  independence  for  red  casualties,  rounds  fired, 
and  exposure  time.  The  engagement  range  variance  was  larger  than  ex- 
pected as  a result  of  independent  experiments. 

Comparisons  were  also  made  when  the  experimental  objective  was  to 
estimate  the  mean  value  of  the  performance  measure.  The  best  result  for 
the  Filter  Model  was  a 35%  reduction  in  cost  to  estimate  expected  red 
casualties  with  equivalent  variances  from  both  blocked  filter  runs  and 
independent  full-length  simulation  runs.  A least-cost  experimental 
design  algorithm  is  presented  in  Appendix  B,  and  this  algorithm  deter- 
mines the  number  of  replications  and  blocks  for  each  system  in  reducing 
the  variance  of  average  performance  for  each  system  below  a specified 
level.  Results  from  this  algorithm  showed  a 25%  reduction  in  cost  to 
estimate  expected  red  casualties,  a 16%  reduction  for  rounds  fired,  and 
a 12%  reduction  for  engagement  range.  The  costs  for  exposure  time 
estimation  appeared  to  increase.  Thus,  the  savings  to  estimate  average 
system  performance  are  less  dramatic  than  estimating  differences  in 
average  system  performance,  but  the  savings  for  estimating  expected 
casualties  are  significant.  For  the  other  performance  measures,  the 
potential  for  savings  in  unclear. 
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APPENDIX  A 


sr 


DERIVATION  OF  ESTIMATORS  FOR  FILTER  MODEL  PARAMETERS 

(HEVCE  Model) 

Introduction 


In  this  appendix,  the  version  of  the  Filter  Model  that  incorporates 
heterogeneous  variances  and  correlated  effects  is  defined  and  unbiased 
estimators  are  derived  for  parameter  values  that  are  important  to  the 
simulation  experimenter.  That  is,  certain  parameter  values  are  required 
to  plan  further  simulation  experimentation  and  to  estimate  the 

(a)  mean  performance  of  each  system  alternative, 

(b)  difference  between  mean  performance  of  a pair  of 
alternatives, 

(c)  variance  of  the  mean  performance  estimators,  and 

(d)  variance  of  difference  estimators. 

The  model  acronym  is  HEVCE  for  HEterogeneous  Variances  and  Correlated 
Effects.  The  Filter  Model  is  specifically  designed  to  represent  important 
characteristics  of  DTNCOM,  a stochastic  combat  simulation,  likely  to  be 
encountered  when  simulation  is  performed  in  blocks.  That  is,  a given 
battle  situation  or  environment  is  obtained,  and  a block  of  replications 
are  observed  (or  simulated)  on  each  alternative.  This  starting  situation 
for  a block  is  called  a block  environment.  Given  a particular  block 
environment  differences  among  the  system  designs  may  be  more  efficiently 
observed  since  a source  of  variation  has  been  removed.  Of  course,  the 
subsequent  results  may  be  related  to  the  particular  block  environment  so 
a number  of  blocks  may  be  required  to  reliably  estimate  system  perform- 
ance measures. 


Filter  Model  Structure 


The  structure  of  the  Filter  Model  is  defined  by  a set  of  assumptions 
concerning  variation  among  system  alternatives,  block  effects,  stochastic 
differences  from  replication  to  replication,  and  interactions  among  these 
effects.  For  a given  replication,  the  following  equation  is  assumed  by 
the  Filter  Model  to  relate  the  principal  effects.  That  is, 

xijk  = H + Bi  + Aj  + ABy  + €ijk  ; (A.l) 

where 

Xi-jk  = observed  system  performance  for  block  i,  system  j , 
and  replication  k; 

H = mean  effect  over  all  systems,  blocks,  and  replica- 
tions ; 
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= main  effect  for  block  i; 

Aj  = main  effect  for  system  j ; 

= interaction  effect  between  block  i and  system  j; 

€ijk  = replication  effect  for  replication  k of  system  j 
during  block  i. 

Caere  are  b blocks  so  i = l,2,«««,b  and  a different  system  alternatives 
so  j = l,2,»»»,a.  In  addition,  nij  replications  are  simulated  for  block 
i of  system  j.  Thus,  k = l,2,««*,nij. 

The  stochastic  properties  of  the  above  variables  and  their  mutual 
relationships  are  important  in  being  able  to  represent  the  results  of 
different  test  plans.  The  usual  experimental  design  models  assume  that 
the  stochastic  variables  are  mutually  independent  and  normally  distri- 
buted (Hicks,  1964,  Graybill,  1961,  and  Winer,  1971) • Significant  ex- 
tensions to  these  assumptions  are  made  in  the  HEVCE  Model.  To  represent 
blocking  effects  in  EffNCCM,  the  following  assumptions  are  made.  The 
parameters  u and  Ajj  j = l,2,««»,a;  are  fixed  effects  or  deterministic. 

That  is,  the  mean  performance  for  system  j is  u + Aj.  Since  p is  the 
mean  performance  overall  alternatives, 

a 

j J/j  = 0-  (A. 2) 

The  block  main  effects,  Bj.,  i = l,2,*»»,b,  are  random  since  the  set  of  b 
blocks  are  selected  from  a large  number  of  different  possible  blocks.  Each 
value  of  Bi  is  assumed  to  be  independently  and  identically  distributed 
with  mean  and  variance  given  by 

E(B0  = 0 (A.  3) 

V(Bt)=a|  (A.4) 

The  block-system  interaction  effects,  AB^,  are  also  assumed  to  be 
identically  distributed  random  variables  with  mean  and  variance 

E(ABlj)=0  (A.  5) 

V(ABy)  = 4b  (A.  6) 

Since  the  system  levels  are  fixed  or  deterministic,  then 

a 

2 AB„  = 0 for  all  i (A.7) 

j = l IJ 

but  the  block  levels  are  stochastic  so 

b 

Z AB„  i 0 for  all  j.  (A.8) 

1=1 
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In  addition,  the  block  and  interaction  effects  for  a given  block  are 
correlated  with  covariances 


COV(Bit  A By)  = E(Bt  • ABy)  - aBAB; 

(A.9) 

for 

d = 1,2, •••,a 
i = l,2,»»»,b 

also 

COV(ABy,  A^t)=  E(A^  • A^t)=  aABAB; 

(A. 10) 

for 

d = l,2,»»»,a 
i = l,2,»»»,b 

Since 

the  results  among  different  blocks  are  mutually  independent,  the 

COV(Bv,ABi;j ) = E(VABij)  = °» 

(A.  11) 

OOV(ABid,ABvj)  = E(ABi;J  * AByj)  = 0, 

(A.12) 

C0V(ABlt,ABvJ)  = E(ABit*AByj)  = 0, 

(A.13) 

for 

i ^ v;  i = 1,2, •••b; 
v = 1,2, •••b; 

t t d;  d » 1,2, •••a;  and 
t = 1,2, •••a. 

The  replication  effects,  have  unique  distributions 

system  alternative.  Their  means  and  variances  are 

for  each 

E<V = °> 

(A.14) 

v(Sjk> * v 

(A.15) 

for 

i - 

i = l,2,»»»,b;  and 
k = l,2,*»»,n^j. 

These  replication  effects  are  correlated  with  both  the  block  and  inter- 
action effects  for  the  block  being  replicated.  That  is, 

COV(Blt  €yk)  = Ogg j (A.l6) 


COV(ABijt  eijk)  = °ABej  (A.17). 
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COV(ABit,€ijk)  = o-^ej 

(A.18) 

covOV^ijk)  - o 

(A.19) 

COV(ABvj,  <4^)  = 0 

(A.20) 

COV(AByt,  €ljk)  = 0 

(A.  21) 

for  t i j,  v + i,  k = 1,  2,  . . . t «ij; 

j - 1,  2,  . . . , a; 

t 1,  2,  » • • | 

i = 1,  2,  . . . , b;  and 

v=  1,  2,  b. 

Note  that  each  covariance  involving  a replication  effect  either  has 
unique  values  for  each  system  being  replicated  or  is  zero.  Moreover, 
the  replication  effects  within  a block  are  correlated  and  the  covari- 
ances are  related  to  the  system  being  replicated.  That  is. 

cov<V  v=v 

(A.  22) 

for 

m /k,  k=l,2,***,  nij;  m = l,2,*",ni;J 

CW^ijk^itk)  = ^ejt  » 

(A.23) 

for 

k = ly2,***,inin(nij,nit) 

cov^eijk»€itm)  = aejt 

(A, 24) 

for 

m i k,  k = l,2,*»»,nijj  m = 1,2, •••nit  ; 

COV(eijk»evjm^  = 0 

(A. 25) 

for 

k = l,2,*«»,ni;3;  m = l,2,«**,iVj  » 

C°V(eijk»evtm)  = 0 

(A.26) 

for 

k = l,2,***,tti;J;  m = 1,2, • • • ,n^.  ; 
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***  'Tj^ee^5^*^w>>yay 


I 

i 

J 

i 


■where  t / j, 

t = l,2,**»,a; 
j = l,2,»»*,a; 
i ^ v; 

i = l,2,»»«,b;  and 
v ~ l,2,.«»,b. 


The  covariance  a’  represents  correlation  among  different  replications 
of  the  same  system,  i.e.,  J,  and  the  covariance  cr*  . represents  corre- 
lation among  different  replications  of  different  systems,  i.e.,  j and  t. 
Note  that  cr  - represents  correlation  among  the  same  replications  of 
difierent  systems,  i.e.,  J and  t.  Unique  values  for  the  covariances 
among  replication  effects  for  the  same  replication  of  different  systems 
are  assigned.  This  is  done  to  represent  the  effects  of  variance  reduc- 
tion using  identical  random  number  sequences  for  each  system  on  a given 
replication  which  tends  to  produce  a positive  correlation  between  system 
performance  measures  for  the  same  replication,  i.e.,  eijk  and 


Variance  of  Estimator  for  Mean  System  Performance 

The  basic  purpose  of  simulation  experiments  conducted  with  the 
process  defined  above  is  to  estimate  the  mean  performance  of  systems 
with  an  unbiased  statistic  having  the  least  possible  variance.  An  un- 
biased estimator  for  system  mean  performance  is  presented  in  this 
section,  and  then  an  expression  for  the  variance  of  this  estimator  in 
terms  of  the  parameters  defined  above  is  derived. 

Estimator  for  Mean  System  Performance 


An  unbiased  estimator  for  mean  system  performance  of  system  j will 
be  developed  by  determining  the  expected  value  of  and  then  computing 
an  average  value  for  X^^  over  sill  blocks  and  replications  for  a fixed 
value  of  j.  The  expected  value  of  X^^  is 


3 

L 

] 

1 

J 

1 


E(xijk)  = H + 


(A.  27) 


since 

E(Bi  + ABi;j  + f-ijk)  = 0 

by  equations  A. 3,  A. 5,  and  A,l4,  Note  that  n + Aj  is  the  mean  perform- 
ance of  system  j.  Thus,  an  unbiased  estimator  for  n + Aj  is  obtained 
by  averaging  the  valueB  of  X^  over  all  blocks  and  replications  for  a 
fixed  value  of  j.  The  sample_average  for  a fixed  block,  i.e.,  i,  and 
system,  i.e.,  j,  is  noted  by  Xj^.,  where 

Xy.  = ^ *ljk  , (A.28) 

D nij  k=l  ° 


53 


where  >0  for  i = l,2,«*«,b  and  j = l,2,»»»,a,  Averaging  over  all 
blocks,  the  unbiased  estimator  for  system  j performance  is  X.^.  and  is 
defined  as 


1 i niJ 


(A.29) 


where  bj  = number  of  blocks  having  at  least  one  replication  of  system  j ; 
and 

E(X.j.)=H+Aj  (A.30) 

Going  one  step  further,  the  mean  of  X.j.  or  X...  over  all  alternatives 
is  an  unbiased  estimator  of  n.  That  is, 


x...  - lit  I HIT  I xijk  (A-31) 

3=1  nyVo  Jk=1 

and 

E(X...)  = n (A.32) 

because  of  equation  A.  2. 

Variance  of  Mean  System  Performance  Estimator 

Having  obtained  an  estimator  for  mean  system  performance,  an  ex- 
pression for  the  variance  of  this  estimator  will  be  derived  in  this 
section.  First,  the  variance  of  an  individual  observation  or  V(Xijk) 
is  found.  In  deriving  both  variance  expressions,  use  will  be  made  of 
the  fact  that 


V(Y)  = E(y2)  - e2(Y)  (A. 33) 

for  any  random  variable  Y. 

To  use  A. 33  in  finding  V(Xjjk),  the  E(Xijk2)  will  be  determined. 

By  definition 


E(Xi;j|)  - E[(H  + Aj  + Bj  + A By  + ] 

= E[(fi  + Aj)2  + E^+ ABy2  + €y2  + 2[(i  + Aj)(Bj  + ABy  + €yk) 


+ 2Bj(ABy  + €yk)  + 2ABy€yk] 


(A. 34) 
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In  the  above  expression,  note  that 


E[2(n  + Aj)(Bi  + ABjj  + cijit)]  = 0 


since  the  expected  value  of  a product  involving  a constant  and  a random 
variable  with  mean  zero  is  zero.  Substituting  the  above  expression  into 
A. 3^  and  using  Af4,  A.6,  A. 9,  A. 15,  A.l6,  and  A.17,  the  following  ex- 
pression for  E(X^j jc^)  iS  obtained. 

E(xijk2)  = (M-  + Aj)2  + <PB  + + 20^2  + 2aBeJ  + 20^^ 

(A. 35) 

Using  A. 35,  A.33,  and  A.27; 

v(xijk)  = + °^AB  + ^ej  + 2aBAB  + 2aBej  + 2aABej  (A.  36) 


To  derive  an  expression  for  V(X,j.),  the  mean  system  performance 
estimator,  use  will  be  made  of  the  fact  that  results  for  a given  block 
are  independent  of  other  block  results.  Thus, 


V(X.j.) 


7 £ v(V> 

3 n13X) 


(A.  37) 


Also,  equation  A.33  will  be  used  in  determining  V(X. );  thus,  an  ex- 
pression for  Efx^j.2]  must  be  determined.  By  definition, 

niJ 

EtX,,.2!  = E[(i  k^h‘  + AJ  + Bl+AB1j*«l)k))2] 

' E[(l*  + A,  + B,  + AB,,  + ^ kSj^Jkf  ] 

'%+V2  + Bl2  + AB1)2  + (^|;.Jk)2 

+2Bl(ABlJ  + Ht)  IVj'HJk)  + 2ABij("ij  kM€Uk)]  (a>38) 


since  all  cross  products  between  a constant  and  random  variables  with 
mean  zero  vanish. 

Taking  the  expectation  of  the  above  random  variables, 
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(A,  39) 


ntj 


E[XIJ.2]  = ^ */  + a\  + a2Afi  + E[(ij 

+ 2aBAB  + 2oBej  + 2oABej 


To  evaluate  A.39»  the  expression  £ e* vd.ll  be  expanded; 

thus,  L'  ^ k=l  3 / -I 


17 1 y \2n 

1 

r nij  nij  1 

[fej  k=i  *&T\ 

= rT2 

niJ 

Vl  m=l€iJk  ’ Vi 

+ <nlJ2  - nlj>CT'e)] 
" ^ej  + <"«  ' 1)0! 


ej’ 


(A.4o) 


for  riij>0;  i = l,2,**»,b;  j = l,2,***,a.  The  above  result  is  true 

since  the  double  summation  in  A.40  involves  n^  terms  where 

eijkeijm  = €ijk2  niJ2"nid  termfl»  «here 
eijk«idm  8X1(1  ® ^ k* 

Using  A. 22  and  A.  15,  equation  A.40  follows.  Substituting  A.40  into  A. 39, 


E[X,j.2]  - 0*+A,»2  + + a2AB  + 2obab  + 2eBeJ 


+2a 


A Be)  + n^e)  ' <“U  ‘ l)a'ej> 


(A.41) 


for  nij>0»  1 = 1*2, • • • ,t» ; j = l,2,»»»,a.  Using  A. 33,  A.27, 

A.4l,  we  obtain 


) ~ fj  [^B  + (l2AB  + 2aBAB  + 

nij>0 


2 a 


Bej 


+ 2nABe)  ^e)  + <»lj  ' D^’ej)] 

bj  (n2B  + ^AB  + 2oBAB  * 2aBe)  + 2aABe)  + “'ej* 

+ 1 (o2  - a'  .)  S — 

b?  ' ®J  er  i,  »ij 

nij>0 


(A.  42) 
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Note  that  from  the  viewpoint  of  analyzing  an  experimental  plan,  the 
above  expression  for  the  variance  of  the  estimator  of  system  J mean  per- 
formance can  be  regarded  as  the  function  of  just  two  parameters.  That 
is, 

1.  system  j block  variance,  V^j,  where 

Vb3  = o®,  + ^ * 20-bab  + ®°BeJ  * 2oABeJ  + <r'aJ  <A-43> 

ftw 

2.  system  j replication  variance,  Vrj,  where 

Vrj-^’ej  <A-W) 

Knowledge  of  the  values  of  both  Vbj  and  Vrj  permits  calculation  of  the 
mean  value  estimator  variance  for  arbitrary  values  of  the  number  of 
blocks  and  replications  per  block. 


Estimation  of  the  Variance  of  the  Estimator 
of  System  Mean  Performance 

Estimators  of  the  values  for  VLj  and  Vr;»  are  derived  in  this  section. 
The  required  estimators  are  obtained  by  first  estimating  Vrj  and  then  V-^. 
Each  estimator  is  unbiased  in  that  they  are  obtained  by  finding  a statis- 
tic with  an  expected  value  equivalent  to  the  parameter  values  required. 

by  _ 

The  statistic  S ^2^  (Xyjj  - Xy.)^  1b  used  to  estimate  Vrj, 

nij>0 


and  the  expected  value  of  this  statistic  is  derived  below. 


7 

4 k 

nlJ>0 


(Xijk~  Xij 


)*\  = E 7 

i, 

J LnU 


nij  , 2 
kll(  ijk  _2X 


ijkXij* 


+ ^|.2) 


,77  .2 


nij>° 


xijk2  - 


(A.  45) 
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Substituting  A. 35,  A.4l,  A. 43,  and  A. 44  into  the  above  equation,  we  ob 
tain 


r n«  - 2 

E 1,  k=?i  <xUte-  XfJ  > 

Enl)>0 


, Z ♦ A,)2  - vb)  ♦ VrJ) 

nlJ>0 


- t + V + vb) + V”ii)] 

nij>° 


In  order  to  find  an  estimator  for_the  parameter  Vy,  the  expected 
value  of  the  statistic  E is  derivea  below. 


n^X) 


E[  E(  nlj(xU-x-J-)1’ 

E[  lt  n«(xir2-2xu-x-r  + x-j-2) 

Uj>o  J 

kj>° 

(A. 48) 


— — — 2 

To  evaluate  A. 48,  the  expectations  of  V.x-a*  and  X. j.  are  required, 
and  egressions  for  these  quantities  are  derived  below. 


E'xu- ' x-r1=Ej  l ElXu-V 

nsj>0 

= s;EiV)  + r 5,  Eixi)-xsi.] 

“sf0 

sj*l 
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Since  the  random  components  of  Xi^k  and  Xfl..k  are  independent  when  s and 
i are  different  blocks,  J J 

= (n  + Aj)l 2 

after  substituting  A. 43,  A. 44,  and  A.4l  into  the  above  expressions, 


= („  * Aj)2  * i-  (Vb3  + Vrj/ny) 


O-  2 


Next,  the  expectation  of  X.j.  is  found, 

EJX.,.2!  = 5-  f E[X„.  X.,.1 

.1  . 


(A.  49) 


-i  i (<*i+A|)2+5<v»j+ w.) 

“if0 

= (H  + A,)2  + i-  VK.  + .L-  V . S i- 

3 bJ  b,2  rJ  4 n*l 

J n.y>0 

Substituting  A.4l,  A.43,  A.44,  A.49,  and  A.50  into  A.48, 


(A. 50) 


f ~ X- j-)Z]  ~ ? nlj(<|A+  Aj)2  + Vbj  + vrj/nij) 

.V°  J m/o 


-2  + £<VbJ+Vnt)>) 

nlj>0  ^ 


l n*j((l*  + V + Bj  vbj  + bj?  Vrj  l 4 ) 

nlj>0  nsj>0 
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? 


II 


E 


l, 

Lnij>0 


nij  <xij‘ 


+ Vrj(bd 


2 + 


(A.  51) 


The  estimator  for  is  determined  from  A.  51  by  replacing  Vr^  with  its 
estimate,  which  gives  J 


v n 
nij>0 


ij 


) 


2 


1 


■ r 

mik 


4 


A 

If  < 0 as  computed  above,  set  VbJ  - o. 

* 

Since  \a  Is  a variance,  a negative  value  for  has  no  physical 
meaning  and  should  be  adjusted.  The  assumption  is  maae  that  the  true 
value  of  Vjjj  is  a small  positive  number  when  the  solution  for  is 
negative  us^ng  equation  A.  51  and  vrj  in  place  of  Vrj.  When  this  case 
occurs,  zero  is  a better  estimate  of  Vm  than  a negative  number.  Al- 
though the  estimates  using  equation  A, 52  are  improved  by  this  procedure, 
the  unbiased  property  of  the  estimators  lost  and  a small  bias  introduced 
because  the  resulting  estimator  is  either  equal  to  or  greater  than  the 
unbiased  estimator. 


‘ ] 


Note  that  the  variance  of  the  mean  value  estimator  for  system  J's 
performance  can  be  estimated  by  substituting  equations  A. 52  and  A.47 
for  their  respective  parameter  values  into  A, 42.  That  is, 


v(x.3.) 


(A.  53) 


” T 


!i 
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Variance  of  Estimator  for  Differences 
In  Mean  System  Performance 

The  correlations  cited  above  among  block,  interaction,  and  replica- 
tion effects  will  cause  the  estimators,  X.j.  ; j = l,2,»»»,a;  to  be  mutu- 
ally correlated.  Thus,  the_var lance  of  a difference  between  two  system 
performances,  e.g,,  X,j,  - X.^,,  1b  not  simply  the  sum,  V(X.j.)  + 'V(X.t.), 
as  one  would  expect  when  the  two  estimators  are  independent.  This  char- 
acteristic is  very  important  when  comparing  systems  because  the  difference 
between  two  systems  may  be  much  more  important  to  the  analyst  than  their 
absolute  magnitude.  In  general,  positive  correlation  between  the  mean 
value  estimators  will  decrease  the  variance  of  the  difference,  X.j.-X.^., 
below  the  sum,  V(X,j.)  + V(X,-fc. ),  On  the  other  hand,  the  variance  of  the 
difference  will  be  increased  by  negative  correlation, 

_ In  this  section  an  expression  for  the  variance  of  the  difference, 
X.j,-X.t»,  where  j ^ t,  is  derived.  Moreover,  an  unbiased  estimator  for 
the  variance  of  the  difference  is  derived  in  a manner  similar  to  that 
used  in  the  previous  section, 

_ Equation  A,  33  is  used  to  determine  the  variance  of  the  difference 
X,j,-X.-t.,  thus,  an  expression  for  Eftx.j.-X.-t,)2]  is  required.  This 
expectation  is  derived  below, 

E[(X.j.-X.t.)2]  = E[x,.j.2  - 2X.j.X,t.  + X.f2]  (A. 54) 


To  evaluate  the  above  equation,  an  expression  for  E(X.j , • X.t.)  1b 
necessary,  and  this  expression  will  be  developed  as  a result  of  a 
sequence  of  steps  involving  the  computation  of  the  following  expectations; 

1,  E(Xijjt  • X^jj)  , 

2,  E(X1Jk  , Xitm)  where  m { k, 

3,  E(XiJk  • Xit.)  , 

4,  E(Xi6.  • Xit.)  , 

5,  E(X.j.  • Xit, ) , and 

6,  E(X.j.  • X.t.)  . 

The  first  expression  is 

E[Xijk  • Xitk]  = E{(n  + Aj  + Bj  + ABjj  + (n  + At  + Bt  + ABit  + ^nk)  ] 

= (P  + Aj)  (H  + At)  + o-g  + 2<tbab  + ffABAB  + '’’sej  + * ABej 

+ °Bet  + * ABet  + * ejt’ 


6l 


after  deleting  all  terms  involving  the  product  of  a constant  and  a zero 
mean  random  variable  and  applying  A,4,  A.9,  A.10,  A.l6,  A,l8,  and  A,23» 

Similarly,  for  the  second  expression, 

ElXijk’  Xitml  " E(  0*  > Aj  + Bt  + ABy  + *1Jk>  <4  + At  + Bj  + ABlt  + «ltm)  ] 

~ + At>  ( ^ + Aj)  + °-b  + 2irBAB  + crABAB  + °Bej  + tr  ABej 


+ <rBet  + ir  ABet  + <reJt* 


where  use  is  made  of  A,  24, 


(A.  56) 


Both  A.  55  and  A.  56  will  he  used  in  determining  the  third  expres- 


nit 

E'xuk  • V)  “ r-  E'xiJk  • xi0n  1 

it  m=l 


= ^ [<nlf1>  («1+At>»‘VAj»+'rB+2,rBAB+'rABAB 
+ °Bej  + * ABej  + °Bet  + ^ABei  + ^ejt) 


+ (y.  + Af)  fli.  + A.)  +<r2  + 2<r  + <r 

v r B BAB  ABAB 

+ <r  + o \ _ . + c_  + cr  * + cr  * "1 

Bej  ABej  Bet  A Bet  ejt  J 

fti  • A.)  (u.  • A.)  * «r 2 + Zo  + o + 0 + 0- ’ 

1 ' J'  B BAB  ABAB  Bej  ABej 


+ W*ABet  + %jt+<%Jt-<reJt)/“lt 


!f  k « nlt.  Otherwise  if  k > nlt  >0,  then 

ElXyk  • Sifl  * + * A,)  0*  + At)  * e2  + 2e  + , 


BAB  ABAB 


+ <rBej  + ^ ABej  + ‘"Bet  + ^ ABet  + * ejt 


(A. 57) 
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For  the  fourth  expression, 

E[Xlr  • Xu.l  = -±-  D E(X1Jk  • xlt.) 
1 nij  k=l 


' * + AJ>  •*  + A,»  + 'I  + 2-bAB  + 'ABAB+'Bej  + %BeJ 


+ <r  to-'  +7*  * ifjsJt  - 
Bet  ABet  ejt  _ov  . 

max  fyj.  Hjt) 


(A. 58) 


where  nj-j,  nit  > 0. 


The  above  equation  will  be  used  in  constructing  the  fifth  expression. 
That  is, 

E[X.r-  Xj,.]  = ^ S E[xBf'XltJ 

V® 

Recalling  that  all  random  variables  in  different  blocks  are  independent, 


1 - — 


EIXT*xifl  ^EIXij*  ’ %1+  — 


min(bj,  bt)  - 1 


(ji  + Aj)  (ji  + At) 


= Oi  + Aj)  Oi  +At)+  ^(cr|  + 2a 


+ (T  + O’  +0" 

BAB  ABAB  Bej  ABej 


+ (r_  , +<r'Ar,  . + <r  * + 

Bet  ABet  ejt 


^ ejt  " ^ ejt  > 

max(nij,  nu) 


lf  nij»nit  > °« 

In  the  event  either  n^j  or  n^  are  zero. 


(A.  59) 


E[X.j.  • X^.  ] = (|j.  + Aj)  0x  + A^)  if  bj  and  bt  > 0. 


For  the  last  expression, 
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isrx.j.  • X.,.J  - ^ 2 E[S.r  - Xlt.  ] 


"lt>0 


°*  + Aj’  * + At»  + rr  (rB  + 2<rBAB+<rABAB 

J ^ 

+ ffBej  + a ABeJ  + ffBet  + ffABet  + ^ejt  * 


b,  b. 


I 


» * 

<r  - <r  * 

eJ*  ejt 


(A.6>) 


1 1 "ij-4  > o ra“(v  v 

where  bjt  » number  of  blocks  with  > 0 and  nJt  > 0. 

Substituting  A.60  and  A. 50  into  A. 54, 

E[  ^-j-  “ X.t.)2]  = (H  + Aj)2  + -g-  Vbj  + Vrj  S jji-  -2(fi  + Aj)  fti  + At) 


2b..  / 

- -Ji  Lz  + 

bjbt  \ 


"ij 


> + ^BAB  + ° ABAB  + ^Bej  + °ABej  + ^Bet  + °ABet  * * +ejt) 


b.  b 


S <rejt“<rejt 


i.  vbt 


i t nirnit>o  max(IHj.  *Ht> 


+ V , 2 _1 

2 rt  i, 


nit>0  ^ 


= (Aj  • At>2  + vbj + vbt +“i  vrj  * _L 


~3  n >o  ij 

+ -L  v s _I_ 

b?  rt  nit 

t n >0  “ 

it 

2bjt  , 

L I a-2  + 2cr  „ +<r  +cr  + <r ' +<r  +0-'  +o- 

bjb  \ B BAB  ABAB  Bei  ABej  Bet  ^Bet  ejt 


* ) 

eit  / 


bjb, 


r 

i, 

n.A>0 


“ij*  nit 


» * 

<T  - <7 

ejt  ejt 

max  (ny,  nlt) 


(A.61) 
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i 
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Using  A.  33#  the  variance  of  the  difference  becomes 


°lt  / 

Vt  ' 


<r^  + 2ff 

B 


+ (T 


+ <r,,  . + «r 


+ <r_  + <r 


+ tr 


BAB  ABAB  Bej  ABej  Bet  ABet  ejt 


"sps  t 

J ns 


^ ejt  " °ejt 


V0’ 

nit>° 


max 


(nij’  nit> 


(a, 62) 


The  above  expression  can  be  simplified  considerably  if  the  assumption  is 
made  that  each  block  having  observations  for  system  j also  contains  ob- 
servations for  system  t and  vice  versa.  Under  this  assumption  bj  = bt  = 
bj£,  and  after  substituting  A. 43  and  A.44 


*V>  =b~2(2,rAB  + toABeJ  + ^ABet  + + "et  ‘ 21 A BAB  ’ 

1 1 


l/T ' -2 <r*  ) + — „ V , S 

ABet  ejt ' * ? rj  i, 


n 


nij>0 


ij 


ABej 


2 vrf  ; 

b.14.  rt  i n 

it 

nit>0 


2 i,  max(n  V 

jt  n£j»njt>0 


when  bj  = bt  = bjt.  (A.63) 

Both  the  estimation  and  analysis  of  experimental  plans  are  simplified 
if  V(X.,.-X.t.)  is  regarded  as  a function  of  two  parameters  requiring 
estimation  in  addition  to  Vrj  and  Vr^,  That  is,  let 


'djt  - 2oAB  + 2aABeJ  + 2c7ABet  + <3  + a’ef  2°ABAB  ‘ 2<rABe3 

- - 2<T=jt-  “4 


J rr* 

Vrjt  ejt  ejt 


(A.64) 

(A.65) 
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is  referred  to  as  the  block  variance  for  the  difference  between 
system  j and  t,  and  Vpj-t  is  the  replication  variance  for  the  difference 
between  j and  t. 


Estimation  of  Replication  Variance 

The  expected  values  appearing  in  equations  A.  55  and  A.  56  immediately 
suggest  an  unbiased  estimator  for  the  replication  variance.  Note  that 
the  difference 


E[xljk  * xitk]  " E[xijk  • Xltm]  = u'eit  - cr*^  = 


Thus,  an  unbiased  estimate  of  can  be  constructed  by  taking  the  dif- 
ference between  two  sample  averages  where  one  average  consists  of  terms 
having  the  expectation  equal  to  EfX^j^  • Xitk]  and  the  other  average  con- 
sists of  terms  with  an  expectation  or  EpC^  • XitnJ.  The  desired 


statistic  is 


rjt 


T 2 
bJt 


min(niJ.  “it) 


nijt  nit>0 


ml“(“ij.  ntt)  k-1 


Xijk  * Xitk 


nij  "it 


j r vur-^ci»>  £1  V x<«" 

1.  


ij  it 


k f m 


where  bj-^  if  the  number  of  blocks  with  n^n^  > 1;  and  e[yJ,j^.]  = . 

(A.67) 

Finding  an  estimator  for  is  a more  difficult  task.  The 
statistic 

b _ _ 

2 n..  Hjj  ( Xji.  - X.. . ) ( X,  ) 

i=l  J 

is  useful  in  finding  an  unbiased  estimator  for  Hie  expected  value 

of  this  statistic  is  derived  below. 


E 


L i-1 


"ij  "it  < Xij*  ' X,j’)  ( Xit*  “ 30  t*)j 

E (A  ^ ^ * X^*  Xjt*  ” X^'  ** ~ X*J*  Xit*  + X*J*  X,t’^ 
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After  substituting  A.43,  A.58,  A.59,  A.60,  A.64,  and  A.65, 


E £ 2^  n^j  n^t  ( Xjj.  - X.  j. ) ( X^  - X.^.jj 


b / 1 Vrjt  \ 

= t=  "M  nit(^  + Aj>^  + At)  + 1 (Vbj  + vbt  - vdjt)  + — ^ ■ niti  j 

b / V'  ' 

2 * HI  "it  ■ * ■ A))  * - At>  ♦ <Vb,  ♦ Vbt  - vd)t)  + 

b / 1 
+ i=  "ij  + A|)«*  + At)  + 2b“  (VbJ  + Vbt  ' vdjt> 


rjt  \ 


V naJ.Sjt>0  ®ax(nSJ,bBt)  ) 

•( (*,  •»'•)  V 


' H- TT— )S  min(nii»  nit>  + S Vnit\  max(nsj,nflt 

rjtU^T/1=1  ij  Djt\i=l  /\n8j’n8t>0 


+ V' 


for  bj  = bt  = bJt 


(A.68) 
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The  desired  estimator  for  V-.^.  is  found  by  solving  A.68  for  and  re- 
placing Vjjj  and  V-^  with  their  respective  estimators.  Thus, 


AAA 

v...  - V,  . + v.4 

djt  bj  bt 


2b 


1L 


<bjt"1)<  jS  nijnit) 


nij  °it  X*  j*  “ X,f> 


Jt  i 

M* 


I 


I 


+ 2V' 


V2 


rjt  min(nirnit> 


Jjf 


V nitJ 


bjt^bjt*1)  ( 8‘  max(nsj*nsl) 

\nsj*  n8t>0 


for  bj  - bt  = bjt. 


(A.69) 


Using_the  above  equation  for  V<yt  and  given  by  A.67,  estimates 
V(X#;j.-X#t. ) can  be  obtained  by  substituting  v^jt  and  into  A.63. 


of  V^A.4 

That  is 


vpc.j.-x.,.)-^-  vdjt+-jvrj  s -l 

i‘  b1t  -‘in  ”1 


l £ 


— V_.  s Err 

b/ 


+ 2 ¥rt  i,  nit 


Jt  njj>0  niJ  bjt  nit>0 


«£  I 


1 1 
»£  ! 


_ A / 

2V 


s J , for  b = b = b14 . 

bj^  i,  max(nij*nit>  J t jt 

nij’  "it1*0 


(A.70) 
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APPENDIX  B 


LEAST-COST  EXPERIMENTAL  DESIGN  ALGORITHM 


This  appendix  presents  the  algorithm  used  to  solve  for  the  least-cost 
experimental  design  to  achieve  a set  of  estimated  variances  less  than  a 
specified  upper  limit,  i.e.,  Vs,  for  each  system  mean  estimate.  The  algo- 
rithm is  provided  a set  of  estimates  for  the  system  block  variances,  Vbj, 
and  system  replication  variances,  Vrj.  In  addition,  a set  of  blocks  may 
already  have  been  observed,  and,  if  so,  the  algorithm  must  determine  the 
additional  experimentation  in  the  form  of  additional  replications  for  the 
old  blocks  and/or  new  blocks  with  replications.  The  algorithm  considers 
the  expected  cost  to  generate  a block  and  to  perform  replications  within 
a block  in  order  to  reduce  the  estimated  variances  for  each  system  mean 
estimate  blow  the  limit  Vs.  The  costs  provided  as  input  data  to  the 
algorithm  are: 

Cb  = cost  to  generate  a new  block,  and 

Cr  = cost  to  perform  a single  replication  for  one  system, 
in  addition  to  Cfe* 

The  solution  provided  by  the  algorithm  is  expressed  by  the  follow- 
ing variables: 

u* 

bj  = total  blocks  to  be  observed  for  system  j 

b*  = maximum  number  of  blocks  to  be  observed  over  all 
systems,  i.e.,  b*  = max  b^* 

r* 

n^  = minimum  number  of  replications  in  each  block 

b’  = number  of  blocks  already  initiated. 

That  is,  the  additional  replications  for  system  j to  be  observed  in  blocks 
already  initiated  is 

* 

nj  - n^j  if  nj  - n^j  >0,  and 
0 if  nj*  - n^s  0;  where  i = l,2,,*,,b',  and 

u* 

n^j  is  defined  on  page  50.  In  addition,  bj  - b'  new  blocks  must  be 
initiated  for  system  j with  n^  replications  in  each  block. 

Of  course  the  algorithm  can  operate  in  the  absence  of  blocks  that 
have  already  been  initiated.  In  this  case,  b'  is  zero  and  ni;»  = 0 for 
all  i. 

The  algorithm  finds  a least- cost  solution  iteratively  by  determining 
the  conditional  least- cost  solution  given  b blocks  for  all  possible  values 


69 


of  b.  For  each  conditional  least -cost  solution,  the  following  values  are 
generated: 

u 

bj  = total  blocks  to  be  observed  for  system  j given  a 
maximum  of  b blocks  (b^  £ b), 

r 

nj  = minimum  number  of  replications  in  each  block  given 
a maximum  of  b blocks. 

The  lowest  possible  value  of  b is  b'  where  all  additional  replications 
are  performed  on  existing  blocks  and  the  largest  possible  value  of  b 
occurs  when  nj  = 1 for  all  j. 

The  solution  for  the  conditional  least- cost  test  plan  given  b blocks 
makes  use  of  the  decomposition  permitted  into  individual  system  test 
plans  when  the  number  of  blocks  generated  is  known.  That  is,  the  least- 
cost  test  plan  from  a overall  viewpoint  consists  of  the  least-cost  test 
plan  for  each  system  considered  individually  to  achieve  a variance  of 
the  system  mean  estimate  no  greater  than  Vs.  This  decomposition  is  only 
permitted  when  the  total  number  of  blocks  is  known  so  that  tradeoffs  be- 
tween the  total  number  of  blocks  and  replications  for  each  system  are  not 
required. 

The  least  cost  solution  for  n^  is  the  lowest  value  of  n?  satisfying 
the  constraint  giving  the  upper  limit  on  the  estimated  variance  after 
the  experimental  plan  is  executed.  That  is,  application  of  A. 42  speci- 
fies the  choice  of  the  smallest  value  of  nj  satisfying 


vbi  1 

+ — E 

b 


i=l  maxCnpny) 


(B.l) 


A solution  to  equation  B.l  giving  a value  of  nj  is  complicated  by  the 
summation 

*>'  V , 

2 r5 » 

i=l  max(nj,nij ) 

but  the  solution  is  simplified  by  ranking  the  values  of  n^;  i = 1,2,»»», 
b* ; such  that  n£j  > n£+]_}j;  t = l,2,»»*,b';  where  n£j  is  a ranked  value  of 
n^j.  Substituting  these  ranked  values,  B.l  can  be  rewritten  giving 


m 

Z 

t=l 


(b-m)Vrj 

b2.nj 


- V, 


S ’ 


(B.2) 


where  m satisfies  n^j  > n ? > n^+1  j and  nj  is  the  smallest  integer 
satisfying  B.2.  Assuming  that  m is  known,  then 
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$£fc$Si 


n., 


(b-m)Vrj 


*•  1 


in  V^-i 


(B.3) 


hdVB  - bVbj  - £ 


t=l  n. 


rd 

TT~ 

tJ 


if 


9 m V-j 

A,  - wbJ  - t?,  T1  > 0 • 

t-i 


■where  m satifies  > n j > n*+1  ^ , and  the  function  [x]  is  the  greatest 

integer  less  than  X.  In  the  event 


m v . 

" bVbj  ‘ z jpr-  £ °> 
t=l  «t j 


t£en  n?  does  not  exist  and  more  blocks  must  be  generated.  The  value  of 
nj  is  determined  iteratively  using  the  procedure  flowcharted  in  Figure  B.l. 

A special  case  occurs  when  the  solution  for  nj  by  B.3  is  one.  In 
this  case,  the  least  cost  solution  may  not  require  a replication  in  each 
block,  i.e.,  bj  may  be  less  than  b.  Applying  equation  B.2,  the  smallest 
value  of  b^  satisfying 


M + _L_  z Za  » (bJ-VvrJ 

b“  W* 


$ V 


S » 


(B.4) 


bj  5 b',  and 

bj  = number  of  blocks  already  generated 
with  n^j  > 0, 


gives  the  required  conditional  least- cost  solution  when  n^  determined 
by  B.3  is  one.  J 


A flowchart  of  the  algorithm  for  determining  the  least  cost  experi- 
mental design  appears  in  Figure  B.l.  The  algorithm  generates  each 
possible  conditional  least  cost  solution  for  all  permissible  values  of  b. 
The  conditional  least  cost  given  b total  blocks  is  computed  by  the  follow- 
ing equation 


:tb  = °b*b  + 2 •Cr(bJ-b') 
j ~1 
a b' 

+ j=l  i-1  U(n^'nij)  * (nJ"niJ}  * 


where 


Ctb  = conditional  least  cost  of  achieving  the  variance 
constraint  given  b blocks,  and 

U/v\  = the  stop  function  where  U/v\  = I1  if  X>0 
W W Jo  if  otherwise. 


The  least  cost  test  plan  is  determined  by  finding  the  value  of  b*  satis 
fying  C^*  = MinCtb ; and  values  of  bj  and  nj , j = l,2,»**,a;  for  this 

u*  r* 

test  plan  are  designated  as  the  least- cost  values  bj  and  . Each 
possible  value  of  b is  calculated  in  order  to  determine  b*  since  is 
not  a unimodal  function  of  b. 


Figure  B.  1.  —Algorithm  for  Determining  Least  Cost  Solution 
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Figure  B.  1,  continued— Algorithm  for  Determining  Least  Cost  Solution 
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solution  lower  cost  than  others, 
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No 
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Record  the  new  lower 
cost  solution 


ct  - ctb 


-j1  - b? 

r*  r 
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b + l-*b 


yes 


Should  another  block  be 
analyzed,  i.  e. , 
is  CON  = .TRUE.  ? 


(STO^ 
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Figure  B.  1,  continued— Algorithm  for  Determining  Least  Cost  Solution 
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APPENDIX  C 

DERIVATION  OF  ESTIMATORS  FOR  THE  FILTER 
INDEPENDENT  EFFECTS  AND  HETEROGENEOUS 
VARIANCES  MODEL  (HEVIE) 

Model  Description 

A special  case  in  which  the  component  effects  of  the  Filter  model 
derived  in  Appendix  A are  independent  with  heterogeneous  variances  is 
presented  in  this  appendix,  and  estimators  for  the  model's  parameters 
are  derived.  The  model  acronym  is  HEVIE  for  Heterogeneous  Variances  and 
Independent  Effects.  In  this  model,  the  replication  effects,  have 

unique  variances  for  each  system,  o|.,  as  defined  in  Appendix  A,  but 
these  replication  effects  are  independent  of  the  block  main  effects,  Bi, 
and  the  interaction  effects,  AB^ j.  In  addition,  the  block  main  effects 
are  independent  of  the  interaction  effects.  Thus,  the  HEVIE  model  is  a 
special  case  of  the  HEVCE  model  derived  in  Appendix  A,  where 


ctbab 

= 0 

°ABAB 

» 0 

aBej 

= 0 

°ABej 

= 0 

cr" 

= 0 

ABej 

o' 

= 0 

ej 

a'  . . 
ejt 

= 0 

°ejt 

= 0 

for  j = 1,2, •••,&:  and  t = 1,2. •••,».  The  parameters  remaining  in  the 
HEVIE  model  are  erg,  o^g,  and  o|j,  j = l,2,»»»»a. 

The  baBic  purpose  of  simulation  ejqperiments  is  to  estimate  the  mean 
system  performances,  X.4. , the  variance  of  the  estimators,  V(X,-j,),  and 
the  variances  of  the  differences  between  two  system  performances, 
V(X.j.-X.t),  The  expression  for  V(X,j.)  given  by  equation  A.42  of  the 
HEVCE  Model  becomes* 


2 b 


(c.l) 


*In  the  HEVIE  model,  the  assumption  is  made  that  nj.j  > 0 for  i and  j ; 
thus,  bj  = b,  j = l,2,#**,a. 
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after  setting  all  covariances  in  A.42  to  zero.  Thus,  the  system  j block 
variance  for  the  HEVIE  model  becomes 


(C.2) 


and  the  system  j replication  variance  is 


(C.3) 


for  j = l,2,*»*,a.  Note  that  the  block  variances  are  identical  for  each 
system  in  this  case.  The  variance  of  the  difference  between  system 
average  performances,  V(X.  j.-X.-fc. ) is  simply  the  sum 


V(X.r  -X.t.)  = V(X.j.)  + V(X:t.)  (c.4) 


since  the  covariances  all  vanish.  Thus,  the  HEVIE  model  can  be  imple- 
mented if  estimators  of  V-^j  and  Vrj,  j = l,2,»»»,a,  are  obtained.  More- 
over, these  estimators  can  be  used  in  the  procedure  described  in  Appendix 
B to  find  the  least  cost  test  plan. 


Estimators 


Replication  Variances 

Equation  A.47  can  be  used  to  obtain  an  estimator  for  Vr4  which  is 
identical  to  the  estimator  employed  by  the  HEVCE  model.  Thus, 


for  j = l,2,««»,a. 


"ij 

£,  <v- V2 


Note  that  the  estimator  for  V.  is  unbiased. 
Block  Variance 


(C.5) 


The  estimator  for  V^j  takes  a different  form  for  the  HEVIE  model 
than  the  HEVCE  model  primarily  because  the  value  of  V^j  is  identical  for 

each  system.  To  estimate  V-^j  = cr|  + oj^,  the  expectations  for  two 

statistics  not  employed  in  the  HEVCE  model  are  required,  Hxese  statis- 
tics are  commonly  known  as  the  block  sum  of  squares,  i.e., 

b _ _ 2 

2 (Xi..-X, .,)  , and  the 
i=l 
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L 


u 


is 


interaction  sum  of  squares,  i.e., 


b a 
2 

i=l  j=l 


2 2 (X|j'  * Xj.  ■ ■ X,  j>  + X«  • « ) • 


Block  Sum  of  Squares 

The  expected  value  of  the  block  sum  of  squares  can  be  expressed  as 


js  (Xj..  -X...)2j  = e[  a (Xi..)2]-b  e[^(x...)2 


(o. 


6) 


The  expectation  of  X^..2  is  evaluated  below. 


£(Xi..)2j  * e||  i Sj  Tqj-  ^ (11  + Aj  + Bj  + ABy + €jjk)j  j 


= m^)2] 


since  2 A,  = 0 and  2 ABn  = 0 as  expressed  by  equations  A.  2 and  A.  7. 
1=1  j=l  J 

Continuing, 


a 


"lj 


E[(^)2]=7E[a  V + i * 


j=l  ij  k=l 


ijk 


after  deleting  all  products  of  a constant  and  a zero  mean  random  variable, 
and  products  of  two  zero  mean  independent  random  variables.  Thus, 


E 


[ <xi-  • )2  J = 


2 2 1 
+<ri.  + -a 


a cr 


B 


J=1  ”*]  ' 


(0.7) 


since  the  values  of  eijk  are  mutually  independent. 
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Using  the  above  result  for  the  expectation  of  X,..2 

can  be  determined. 


r b b - r b _ ? b b _ _ 

= -1-  El  2 2 X,. . X . • I = El  2 Xj.  , + 2 S 2 Xi,.X  . < 

b2  Li=i  s*i  1 • J ? Li=i  1 i=i  8*i+i  s 


a 

2 


j=l 


b 

2 

1=1 


where  the  relationships 
EfXj..  X8.. 


n 

2 1 = 
1=1 


n(n+l) 

2 


were  used  in  evaluating  the  last  term  in  C.8. 


(C.8) 


Substituting  C.7  and  C.8  into  C.6,  the  expected  value  of  the  block 
sum  of  squares  becomes 


■ b 

E 2 (Xj..  - X... 
1=1 


a b o-2 
2 2 — £ 
j=l  1=1  nij 


-(•*2-2b 


a b 
2 2 
j=l  1=1 


- (b-l)n2 


= <b-l) 


b 

2 

1=1 


(c.9) 


Interaction  Sum  of  Squares 

The  expected  value  of  the  interaction  sum  of  squares  is  derived 
below. 
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rb  a - - - - 2i 

|^2  2 (Xy.  - Xv.  - X.j.  + X...) J 

^ a _2  — 2 - 2 — 2 - — _ _ 

= Ej^  2 2 (Xy.  + Xrr  + X.j.  + X...  - 2^.  Xj..  - 2Xy.  X.j. 

i 

+2Xy.  X.  • • + ZXj, , X,  j,  - 2Xj, , Xi . . — 2X.  jt  X. . . ) 

■ 


after  expanding  the  expression  squared, 
r b a r b 


= e|s  2 Xtj.2j+ E^  2 aXit.2j  + e|  S bX.j.2J  +ab  E^X..?  j 
+ E £ 2 (-2a  5q. 2 + 2a  5^. . X...  +2aXj..  X...  -2a5ii..2)j 


f 2 (-2b  X.  y - 

L 1=1  J 


2b  5C..  T) 


= E £ 2 2 Xy.2J  - a E £ 2 5q..2|-bE^S  X.^2  + abE^X...2J 


Using  equation  A.4l  and  noting  that 


°BAB  " °Bej  “ °ABej  " aej  " 0 


(C.10) 


for  the  HEVIE  model,  then 


E [Xlr2  ] 


(^+Aj)Z+»J+TAB+T5L  * 


Similarly  from  equation  A, 50,  we  obtain 


EIX.J.2!  - +-AB»+ 


(c.ll) 


(C.12) 


8l 


In  using  the  above  equations  to  evaluate 


f a 
S 

j 


“ _ 2]  r a _ 2l 

* V J and  E[jf1  XT  J 


, the  constraint  2 AB,,  = 0 should 

j=l 


be  rec; gnized. 

In  evaluating  E 
straint  so  that 


E[Jl  ab1j]  = <a- 


a degree  of  freedom  is  lost  to  the  con- 


°AB 


Substituting  these  relations  into  C.10,  we  have 

r b a - - - _ zi 

£22  (X...  - Xv.  -X...  +X...  ) 

L 1=1  j=l  J J 


a 

= S 

j 


S b ( n + Ajr  + abo-g  + (a  - 1)  b + S^  <rej  S^  - 
-ab(|x2+«r|,-  \ - £ b<>i  +A/ 


i) 


2 -i Z ,2  l 1 


” a aB  '<a-l>  aAB  -b  £ ^ 


a 2 b 1 ...  2 


2 (a-l)(b-l)  * 2 b 1 

= (a-  1)  (b-  1)  «rAB  + L-^-L  = °ej  £ ^ 


(c.13) 


Block  Variance  Estimator 


The  expected  value  expressions  for  the  block  and  interaction  sum  of 
squares  derived  above  provide  the  basis  for  an  unbiased  estimator  of  the 
block  variance  + aj^.  Using  C.13  and  C.9,  we  can  solve  for  the 
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expected  value  of  Vbj.  After  replacing  parameter  values  with  unbiased 
estimators  and  removing  expectation  symbols,  then 

b _ 

vbj  = ?b2+^ab  = s -x—r/ft-1) 


+ s s (Xli.  -5q..  -5T...  + X...  )2/[(a-i)(b-i)] 
i-i  j=i  J 1 

a b 

. ft'1)  s £ 2 S _i_ 

azb  j=i  ei  i=l  "lj  (C.l4) 


Note  that  C.l4  is  an  unbiased  estimator  for  Vbj j however,  negative  esti- 
mates may  occur  in  practice  for  Vbj  when  the  estimates  for  Vrj  are 
sufficiently  large  and  the  interaction  and  block  sum  of  squares  are 
sufficiently  small.  Of  course,  a negative  value  of  vbj  has  no  practical 
meaning,  so  a procedure  similar  to  that  employed  by  the  HEVCE  model  1b 
used.  Thus,  Vbj  will  be  used  fran  C.l4  if  it  is  positive;  otherwise, 

Vbj  is  set  to  zero.  The  assumption  is  made  that  the  true  value  of  Vbj 
is  likely  to  be  negligible  when  negative  values  from  C.l4  occur.  A1-a 
though  this  procedure  will  yield  better  estimates,  the  estimator  for  Vbj 
that  deletes  negative  values  is  biased. 


APPENDIX  D 


DERIVATION  OF  ESTIMATORS  FOR  THE  FILTER  INDEPENDENT 
EFFECTS  AND  HOMOGENEOUS  VARIANCES  MODEL  (HOVIE) 


Model  Description 

This  appendix  presents  the  Filter  Independent  Effects  said  Homogene- 
ous Variances  Model  (HOVIE),  and  estimators  for  model  parameters  are 
derived.  The  HOVIE  model  amounts  to  a farther  simplification  of  the 
HEVIE  model  presented  in  Appendix  C,  and  the  HOVIE  model  is  equivalent 
to  mixed  two-factor  model  commonly  found  in  experimental  design  litera- 
ture (Hicks,  1964;  Graybill,  1961;  and  Winer,  1971).  The  HOVIE  model 
is  obtained  from  the  HEVIE  model  by  setting 

<rf.  = erf  for  all  j.  (D.l) 


Thus,  the  HOVIE  model  specifies  that  all  systems  have  the  same  variance, 
and  it  follows  that  they  would  have  the  same  test  plan. 

The  variance  of  the  estimator  for  system  j mean  performance  is 


V<*T>  = b 


(D.2) 


which  is  obtained  directly  from  C.l.  As  specified  for  the  HEVIE  model 
by  C.2,  the  system  j block  variance  for  the  HOVIE  model  is 


(D.3) 


In  addition,  the  replication  variance  is 


Vrj  = cr2  , for  j = 1,2,*.., a.  (D.4) 

As  done  by  the  other  models,  the  least -cost  experimental  design  can  be 
prepared  given  estimates  of  V^j  and  Vrj,  and  these  estimates  can  be 
used  in  the  procedure  described  in  Appendix  B to  prepare  the  experi- 
mental plan. 
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Parameter  Estimators 


Estimators  for  and  are  specified  in  this  section  using 
results  developed  for  the  HEVIE  and  HEVCE  models. 


Both  the  HEVIE  and  HEVCE  models  use  the  result  specified  by  equa- 
tion A. 46  in  deriving  an  unbiased  estimator  for  Vrj  . That  is, 


E 


- 2 

S 2 <Xljk-Xir) 
i,  k=l 


nij>0 


= V 


rj 


t 


2 

i. 

.V0 


Hj " DJ 


Since  the  replication  variances  for  each  system  are  identical  for  this 
model,  then  the  statistic  cited  above  can  be  summed  over  each  system  to 
provide  an  estimator  for  VrJ  = o2j  j = l,2,**»,a.  Ihe  expectation  of 
this  statistic  is*  e 


r , n.. 

b a ij 

2 2 2 

,1=1  j=l  k=l 


<Xijk  “ 


"1  r b 

» 

' = 2 

J Li=1 


(D.5) 


Thus,  an  unbiased  estimator  for  Vrj  is 

A banij  / r b a i 

vrj  - 2 S s (X,jk  - X„.  )2/  S 2 njj-ab  U;2.  (d.6) 

1=1  j=l  k=l  / Li=i  |f1  J e 


a The  same  estimator  as  employed  by  the  HEVIE  model  can  be  used  for 
Vbj  after  accounting  for  the  fact  that  Vr1  is  a constant  function  of  j. 
Thus,  0 


VbJ  “ ^n+(rAB  “ 2 (Xi**  _ X. . . )2/(b-l) 

J a 1=1 


+ 22  (X...  -X...  - X.,.  + X...  )2/[(a-l)(b-l)] 
1=1  j=l  J 


£21  s l 

a2b  6 )=1  1=1  "lj 


(D.7) 


*A11  values  of  n^j  are  assumed  to  be  greater  than  zero  in  the  HOVIE 

model. 
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ID. 7 is  an  unbiased  estimator  for  Vu^,  but  again,  adjustments  may  have  to 
be  made  in  the  event  Vbj  is  less  than  zero.  As  specified  for  the  HEVIE 
and  HEVCE  models,  negative  values  of  Vbj  are  set  to  zero  giving  more 
| realistic  estimates  but  removing  the  unbiased  property  for  vbj. 
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APPENDIX  E 


FILTER  PROGRAM 


Description 


Purpose 

The  Filter  program  accepts  data  generated  by  blocked  simulation 
replications  and  estimates  Filter  Model  parameters.  These  parameters 
are  used  to  estimate  the  variance  of  average  system  performance  for 
each  system  alternative  and  the  variance  of  the  difference  in  average 
performance  between  each  possible  pair  of  system  alternatives.  Finally, 
a set  of  least-cost  test  plans  are  determined  so  that  the  variance  of 
each  Systran's  average  performance  is  less  than  a specified  upper  limit 
VB.  A sequence  of  values  for  Vs  are  considered,  an  a test  plan  is  de- 
termined for  value  of  VB. 


Input  Data  Format 


Card  Number 

Variable 

Columns 

Format 

1 

Title 

1 to  60 

Literal 

characters 

2 

Title,  continued 

1 to  60 

Literal 

characters 

3 

NS 

1 to  5 

integer 

3 

BP 

6 to  10 

integer 

3 

NVARS 

11  to  15 

integer 

3 

VARI 

16  to  25 

real 

3 

DVARS 

26  to  35 

real 

3 

CR 

36  to  45 

real 

3 

CB 

46  to  55 

real 

4 

N(l,l) 

1 to  5 

integer 

4 

• 

N(l,2) 

• 

6 to  10 

• 

integer 

• 

• 

• 

4 

• 

N(1,NS) 

• 

4ns  + 1 to  5NS 

• 

• 

integer 

4 

• 

N(2,l) 

• 

5NS  + 1 to  5 (NS+1) 

• 

integer 

• 

• 

3 + NN 

• 

N(BP,NS) 

• 

• 

• 

• 

• 

integer 

(Continue  entering  the  N array  by  rows  starting  with  the  first  row,  then 
the  second  row.  Each  value  is  entered  in  a field  of  length  five  columns, 
and  NN  cards  are  used  with  eighty  columns  available  on  each  card.) 
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Card  Number 

Variable 

Columns 

Format 

4+NN 

X(l,l,l) 

1 to  10 

real 

4+nn 

X(l,l,2) 

11  to  20 

real 

4+nn 

• 

X(l,l,3) 

• 

21  to  30 

• 

real 

• 

• 

4+nn 

• 

X(1,1,N(1,1)) 

• 

10  i(l,l)-9  to 
10  N(l,l) 

• 

• 

real 

5+NN 

X(l,2,l) 

1 to  10 

real 

5+NN 

/ _ 

X(l,2,2) 

11  to  20 

real 

(Continue  entering  observation  values  for  the  first  block  and  second 
system.  Up  to  eight  entries  can  be  made  on  a single  card.  If  more 


than  eight  observations  have  been  recorded  for  a system,  use  more  than 
one  card.  Always  start  a system  on  a new  card.  After  recording  the 
first  block,  enter  the  observations  for  the  second  block  using  the  same 
format. ) 

Definition  of  Variables 


BJ(  J) 

BP 

BU(j) 

CB 

CR 

DVARS 

N(I,J) 

NR(j) 

NS 

NVARS 

VARI 

VARS 

vbe(j) 

vdbe(j,k) 

VDPS( J,K) 

vr(  j) 

VXBAR(J) 

VXBDIF 

XBJJ 


Number  of  blocks  having  replications  of  system  J 

Number  of  blocks  already  run 

Total  blocks  to  be  run  for  system  J 

Cost  to  run  a block 

Cost  to  perform  one  replication  of  one  system 
Decrease  in  value  of  Vs  for  each  case  considered 
Number  of  replications  already  run  for  system  J 
in  block  I 

Number  of  replications  for  system  J to  be  run  in 
each  block 
Number  of  systems 

Number  of  different  values  of  Vs  considered  (number 
of  different  test  plans  computed) 

Initial  value  of  VB  considered  (each  successive 
value  is  decreased  by  DVARS) 

Upper  limit  on  variance  of  system 
System  J block  variance  (v^j) 

Block  effect  variance  for  estimated  difference 
between  performance  of  systems  J an  K (v^jt) 
Replication  effecj  variance  for  the  difference 
between  J and  K (VyJk) 

System  J replication  effect  variance 
Estimated  variance  of  the  mean  performance  of 
system  J (v(X,j.)) 

Variance  of  the  estimated_difference  between  the 
performance  of  J and  K($(X,  j.-X.^, )) 

Estimated  difference  between  the  mean  performance 
of  system  J and  JJ 


Method 


1.  Read  NS,  BP,  NVARS,  VARI,  DVARS,  CR,  CB 

N(I,  J)  for  J « 1,  NS  and  I » 1,  BP 

X(I,  J,  K)  for  k = 1,  N(I,  J),  J = 1,  NS,  and  I “ 1,  BP 

2.  Compute  XJ(J)  = X.j.  for  J=  1,  NS 

3.  Compute 

VR(J)  by  equation  A.  47 
VBE(J)  by  equation  A.  52 
VXBAR(J)  by  equation  A.  53 

for  J = 1,  NS 

4.  Compute 

VDPS(J,  JJ)  by  equation  A.  67 
VDBE(J,  JJ)  by  equation  A.  69 
VXBDIF  by  equation  A.  70 

for  J * 1,  NS, 

JJ  = 1,  NS,  and 
Jjf  JJ 

5.  Set  VARS  = VARI 

IV  = 1 

6.  Determine  the  least  cost  test  plan  to  reduce  the  estimated  variances  for 
each  system  performance  measure  below  VARS.  Use  the  procedure 
specified  in  Append  be  B. 

7.  If  IV  = NVARS,  go  to  step  9 

8.  IV»IV+1 

VARS  = VARS  - DVARS 
Go  to  step  6 
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